- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies whether frozen video diffusion features support motion perception that is decoupled from camera movement. In particular, a decoder on top of a frozen Wan2.2 backbone predicts moving object segmentation (MOS) and the magnitude of world-space 3D displacement (MME). Experiments on FBMS, SegTrack v2 and DAVIS for MOS, and on MOVi-E, Spring and Dynamic Replica for MME, report improvements over existing methods. It further claims that video diffusion features transfer better from synthetic to real videos than DINOv2 and VGGT.
Critical strengths
-
The idea of probing frozen video diffusion features for scene motion that is decoupled from the camera is interesting and well motivated. Separating clip-level moving-object extent from instantaneous 3D motion magnitude is also reasonable.
-
Competitive MOS results are reported without relying on SAM or optical flow, which reduces the dependence on external modules.
Critical weaknesses
-
Lack of controlled experiments to support the central claim. The paper concludes that "this capability is rooted in pretrained video diffusion features rather than decoder training alone" (L024-025). However, no decoder-only or randomly initialized backbone result is reported. It seems that the decoder itself is somewhat complex, with a temporal CNN, a 4-layer ViT and an RGB encoder. The comparison in Tab. 3 is also uneven. DINOv2 and VGGT use a modified temporal decoder (App. B.1) and get no layer sweep, unlike Wan2.2. Besides, it would be better to evaluate a self-supervised video backbone that is not trained for generation, e.g., V-JEPA 2 [ref 1], as a control. I wonder whether the benefit comes from diffusion training or from video pretraining in general. It is highly recommended to add these controls before attributing the ability to video diffusion features.
-
Unconvincing MME comparison with 4D reconstruction models. Under cMAE, predicting zero motion everywhere gives an error of exactly 1.0. Yet MotionCrafter is above 1.0 on all three datasets, and 4RC reaches 1.252 on Spring (Tab. 2), i.e., worse than a static prediction. This suggests problems in how the baseline outputs were converted, or in the metric itself. I suggest reporting trivial baselines, e.g., zero and mask-constant predictions, and a simple pipeline combining depth, camera pose and optical flow. Separate errors on static and dynamic regions would also help. Without these, it is difficult to fully assess the MME advantage over 4D reconstruction models.
- [ref 1] Assran, Mido, et al. "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning." arXiv:2506.09985 (2025).
Questions for authors
-
How were layer 30 and =0.25 selected? Fig. 5 sweeps them on FBMS and MOVi-E, which are also reported benchmarks, so a held-out selection split would make the comparison fairer.
-
It is arguable whether this question "has not been directly examined" (L121-122). MOFT [ref 2] already showed that frozen video diffusion features contain motion-aware channels that can be extracted without training. It might be better to clarify how MOS and MME go beyond it.
-
What are the decoder size, the peak GPU memory of the 14B backbone, and the variance across training seeds for Tabs. 1-3?
- [ref 2] Xiao, Zeqi, et al. "Video Diffusion Models are Training-free Motion Interpreter and Controller." NeurIPS 2024.
4 comments
Response to Reviewer u3xd (Comment 2/4)
Decoder comparability and feature-layer selection
Q2a:
DINOv2 and VGGT use a modified temporal decoder (App. B.1) and get no layer sweep, unlike Wan2.2.
[A2a]: Thank you for this helpful suggestion. Our initial use of final-layer features was consistent with SegAnyMo’s choice for DINOv2 and GMOS’s choice for its geometry backbone. We used a single feature layer per backbone to keep feature budgets comparable, accounting for differences in temporal resolution and channel dimensionality.
Regarding the temporal decoder, the adaptation in App. B.1 accommodates the different temporal resolutions of the backbone features. Temporally compressed diffusion features require temporal upsampling, whereas DINOv2 and VGGT provide frame-level features and use a comparable spatiotemporal CNN without temporal upsampling. This design retains temporal modeling while keeping trainable parameter counts similar, as described in the appendix. Overall, these changes only adapt the decoder to the different temporal dimensions of the inputs; they do not change the decoder architecture.
Following your suggestion, we evaluated L11, L17, and L23 for both DINOv2 and VGGT, using a layer-search strategy similar to that used for Wan2.2. These experiments confirm that the final layer is not always optimal. We now use DINOv2 L17 and VGGT L17 global-attention features, keeping the selected layer fixed across tasks and training settings. The updated results for Table 3 are shown below.
| Backbone | Mixed FBMS J&F ↑ (%) | Mixed MOVi-E cMAE ↓ | Synthetic-only FBMS J&F ↑ (%) | Synthetic-only MOVi-E cMAE ↓ |
|---|---|---|---|---|
| DINOv2 (L17) | 86.6 | 0.496 | 66.1 | 0.467 |
| VGGT (L17, global) | 83.7 | 0.466 | 55.1 | 0.395 |
| V-JEPA 2 (L39) | 88.7 | 0.428 | 73.9 | 0.381 |
| CogVideoX-5B | 83.4 | 0.465 | 77.5 | 0.403 |
| Wan2.1-T2V-1.3B | 86.9 | 0.441 | 80.4 | 0.387 |
| Wan2.2-I2V-14B | 90.9 | 0.396 | 86.7 | 0.341 |
We additionally compared global- and frame-attention features at VGGT L17.
| VGGT L17 features | Mixed FBMS J&F ↑ (%) | Mixed MOVi-E cMAE ↓ | Synthetic-only FBMS J&F ↑ (%) | Synthetic-only MOVi-E cMAE ↓ |
|---|---|---|---|---|
| Global | 83.70 | 0.4658 | 55.10 | 0.3949 |
| Frame | 84.76 | 0.4479 | 42.93 | 0.3989 |
We retain global features as a consistent choice across settings, supported by their stronger cross-domain transfer in this comparison.
Although layer and feature selection improve the baselines, Wan2.2 outperforms all tested DINOv2 and VGGT configurations across the four evaluation settings. We will incorporate these findings into Table 3 and the corresponding discussion in the revised manuscript.
Self-supervised video-pretraining control
Q2b:
Besides, it would be better to evaluate a self-supervised video backbone that is not trained for generation, e.g., V-JEPA 2 [ref 1], as a control. I wonder whether the benefit comes from diffusion training or from video pretraining in general.
[A2b]: Thank you for this suggestion. We have added V-JEPA 2 as a self-supervised video control and used the same independent Kubric validation set as for Wan2.2 to select its feature layer. We evaluated L20, L30, and L39, and report the selected L39 consistently across tasks and training settings in the table above.
We agree that video pretraining is an important source of this capability, and do not claim that it is exclusive to diffusion models. Our motivation for studying video diffusion models is that they are among the most extensively scaled video-pretrained models, while their dense reconstruction objective encourages the preservation of fine-grained, low-level details, such as object boundaries, textures, and small local changes. These properties make them promising representations for motion readout.
Response to Reviewer u3xd (Comment 3/4)
cMAE, trivial baselines, and reconstruction-output conversion
Q3:
Unconvincing MME comparison with 4D reconstruction models. […] This suggests problems in how the baseline outputs were converted, or in the metric itself. I suggest reporting trivial baselines, […] and a simple pipeline combining depth, camera pose and optical flow. Separate errors on static and dynamic regions would also help.
[A3]: Thank you for suggesting these diagnostics. An all-zero prediction indeed has cMAE = 1 on clips with nonzero GT motion. However, predicting zero everywhere misses actual scene motion. The complementary e15 metric in Table 2 measures the percentage of valid pixels whose absolute magnitude error exceeds 15% of the clip's maximum GT motion magnitude. Zero prediction yields e15 values of 5.32%, 3.69%, and 14.26% on MOVi-E, Spring, and Dynamic Replica, respectively, which are worse than those of MotionCrafter, 4RC, and our method on all three datasets.
Following your suggestion, we additionally report zero-motion, all-one, GT mask-constant, RAFT-magnitude, and depth–pose–flow baselines. GT mask-constant predicts one on GT-dynamic pixels and zero elsewhere before scale alignment, providing an oracle test of whether identifying moving regions alone is sufficient. Depth–pose–flow combines VGGT-predicted depth, camera intrinsics and poses with RAFT optical-flow correspondences. It back-projects corresponding pixels into a common world frame and computes their 3D displacement magnitudes, without using GT geometry or motion masks to generate predictions.
To distinguish motion estimation from static-region suppression, we report errors separately on dynamic (nonzero GT displacement) and static (zero GT displacement) pixels. Both regions use the same prediction-to-GT scale fitted on all valid pixels; static error is reported as MAE rather than normalized by zero GT motion.
When evaluated only within dynamic regions, both MotionCrafter and 4RC achieve cMAE below 1 on all three datasets. Their larger static-region errors indicate weaker discrimination between moving and stationary regions. Predicting nonzero motion over large static regions can substantially increase full-image cMAE, while having a smaller effect on e15 when these residual predictions remain below its error threshold. Please see Figure 4 and Appendix Figures 7 (Dynamic Replica) and 8 (DAVIS) for qualitative comparisons illustrating these residual background responses and our more selective motion predictions.
Our method achieves substantially lower static-region errors than both reconstruction baselines across all three datasets, together with lower dynamic-region cMAE in most comparisons. Within dynamic regions, it also outperforms all trivial baselines on all three datasets. Together, these results demonstrate strong local, instantaneous moving/static discrimination and the ability to estimate motion intensity beyond a binary moving/static distinction.
MOVi-E
| Method | Full cMAE ↓ | Full e15 ↓ (%) | Dynamic cMAE ↓ | Static MAE ×1000 ↓ |
|---|---|---|---|---|
| Zero motion | 1.0000 | 5.32 | 1.0000 | 0.0000 |
| All-one | 1.7461 | 4.69 | 1.0337 | 4.0259 |
| GT mask-constant — oracle | 1.3277 | 5.28 | 1.3277 | 0.0000 |
| RAFT magnitude | 1.7595 | 2.65 | 0.6967 | 4.3342 |
| Depth–pose–flow (VGGT + RAFT) | 1.0489 | 5.48 | 0.9644 | 0.2841 |
| ────────────────── | ──────── | ──────── | ──────── | ──────── |
| MotionCrafter | 1.3043 | 2.01 | 0.5741 | 3.1283 |
| 4RC | 0.4534 | 1.14 | 0.2833 | 0.5205 |
| Ours | 0.3522 | 1.11 | 0.2809 | 0.0933 |
Spring
| Method | Full cMAE ↓ | Full e15 ↓ (%) | Dynamic cMAE ↓ | Static MAE ×1000 ↓ |
|---|---|---|---|---|
| Zero motion | 1.0000 | 3.69 | 1.0000 | 0.0000 |
| All-one | 1.7687 | 2.57 | 0.8968 | 0.6860 |
| GT mask-constant — oracle | 0.7324 | 1.07 | 0.7324 | 0.0000 |
| RAFT magnitude | 1.3901 | 1.60 | 0.7216 | 0.3930 |
| Depth–pose–flow (VGGT + RAFT) | 1.0820 | 3.43 | 0.9735 | 0.0855 |
| ────────────────── | ──────── | ──────── | ──────── | ──────── |
| MotionCrafter | 1.7305 | 1.32 | 0.6637 | 0.6105 |
| 4RC | 1.2516 | 1.80 | 0.7136 | 0.2675 |
| Ours | 0.7569 | 1.23 | 0.6606 | 0.0375 |
Dynamic Replica
| Method | Full cMAE ↓ | Full e15 ↓ (%) | Dynamic cMAE ↓ | Static MAE ×1000 ↓ |
|---|---|---|---|---|
| Zero motion | 1.0000 | 14.26 | 1.0000 | 0.0000 |
| All-one | 1.4872 | 10.73 | 0.8118 | 1.5184 |
| GT mask-constant — oracle | 0.7732 | 10.60 | 0.7732 | 0.0000 |
| RAFT magnitude | 1.0447 | 8.16 | 0.6071 | 0.8513 |
| Depth–pose–flow (VGGT + RAFT) | 0.9728 | 13.28 | 0.9018 | 0.1432 |
| ────────────────── | ──────── | ──────── | ──────── | ──────── |
| MotionCrafter | 1.3767 | 11.69 | 0.7692 | 1.3089 |
| 4RC | 0.5874 | 5.36 | 0.4547 | 0.2052 |
| Ours | 0.5362 | 6.33 | 0.5214 | 0.0351 |
Response to Reviewer u3xd (Comment 4/4)
Selection of layer 30 and σ=0.25
Q4:
How were layer 30 and [σ]=0.25 selected? Fig. 5 sweeps them on FBMS and MOVi-E, which are also reported benchmarks, so a held-out selection split would make the comparison fairer.
[A4]: Layer 30 and σ≈0.25 were selected on a held-out subset of our own Kubric-based synthetic dataset, separate from decoder training and the reported benchmarks. The selection set contains 100 scenes, approximately 10% of the size of our original synthetic dataset.
We evaluated all 12 combinations of layers {20, 30, 39} and noise levels σ≈{0.003, 0.25, 0.50, 0.75}, using the same eight-epoch decoder-training budget and selecting the configuration with the lowest cMAE. Layer 30 and σ≈0.25 achieved the best cMAE of 0.3965, followed by layer 39 at the same noise level with 0.4162. Fig. 5 presents benchmark sensitivity analyses; it was not the basis for hyperparameter selection. We will clarify this selection protocol in the revised manuscript.
| Noise σ (approx.) | L20 | L30 | L39 |
|---|---|---|---|
| 0.003 | 0.5095 | 0.4560 | 0.4691 |
| 0.25 | 0.4347 | 0.3965 | 0.4162 |
| 0.50 | 0.4353 | 0.4164 | 0.4367 |
| 0.75 | 0.5198 | 0.4632 | 0.4881 |
Relationship to MOFT and scope of the novelty statement
Q5:
It is arguable whether this question "has not been directly examined" (L121-122). MOFT [ref 2] already showed that frozen video diffusion features contain motion-aware channels that can be extracted without training. It might be better to clarify how MOS and MME go beyond it.
[A5]: Thank you for pointing out this relevant connection. We agree that MOFT has demonstrated motion-aware channels in frozen video diffusion features and their utility for motion transfer. We appreciate your suggestion to clarify how MOS and MME go beyond this finding.
- Scene motion independently of camera movement. MOFT studies motion broadly, encompassing both camera and object motion for motion transfer. We specifically evaluate scene motion with camera motion excluded from the target, through MOS at the object and clip level and MME at the local, instantaneous level.
- Direct decoding of motion information. MOFT analyzes which internal representations are associated with motion and applies them to motion transfer. We investigate whether motion information in frozen representations can be explicitly decoded into moving-object masks and local motion magnitudes, with accuracy evaluated directly against ground-truth annotations.
We will explicitly discuss MOFT and clarify these distinctions around L121–122.
Decoder size, GPU memory, and seed variance
Q6:
What are the decoder size, the peak GPU memory of the 14B backbone, and the variance across training seeds for Tabs. 1-3?
[A6]: Thank you for requesting these details.
Our decoder contains 74.30M trainable parameters. For the 14B backbone, frozen feature extraction through block 30 requires 26.46 GiB peak allocated GPU memory on a single GPU using BF16 precision, batch size 1, and a 48-frame input at 480 × 848 resolution. This measurement includes the VAE and the retained backbone blocks.
For the main model, we repeated the complete 15-epoch decoder training with three independent training seeds (42, 1, and 2), keeping the backbone frozen and the training and evaluation protocols fixed. The results are reported as mean ± sample standard deviation below.
| Dataset / metric | Raw model | With SAM 2 propagation |
|---|---|---|
| FBMS J&F ↑ | 92.10 ± 0.22 | 93.00 ± 0.22 |
| SegTrack v2 J&F ↑ | 84.01 ± 0.32 | 86.90 ± 0.39 |
| DAVIS 2017-M J&F ↑ | 87.92 ± 0.27 | 90.71 ± 0.38 |
| MOVi-E cMAE ↓ | 0.355 ± 0.004 | — |
| Spring cMAE ↓ | 0.762 ± 0.008 | — |
| Dynamic Replica cMAE ↓ | 0.538 ± 0.002 | — |
Response to Reviewer u3xd (Comment 1/4)
Thank you for recognizing the motivation and the complementary object-level and point-level readouts, and for identifying concrete controls needed to strengthen the evidence. We address the attribution, evaluation, selection, related-work, and reproducibility concerns below.
Contribution of pretrained features versus decoder training
Q1:
[A1]: Thank you for this suggestion. We retrained the decoder with clean VAE latents or Gaussian noise replacing the DiT features, while retaining the RGB branch and using the same eight-epoch training budget for all three inputs. Both controls substantially underperform pretrained DiT features on MOS and MME (MOS without SAM2).
Fig. 5 and Table 3 provide complementary evidence: performance varies substantially with the feature layer, noise level, and backbone despite comparable decoder training. Together, these results show that decoder capacity alone does not explain the strong MOS and MME results; informative pretrained features are essential to accurate motion readout.