acceptodds
Under review as a conference paper at ICLR 2027

Reading Out Camera-Decoupled Motion from Frozen Video Diffusion Features

Abstract

Understanding scene motion independently of camera movement is fundamental to robotics, autonomous navigation, and video understanding. Apparent motion in monocular video, however, entangles scene dynamics with camera movement. Video diffusion models generate coherent object motion under diverse camera movement, raising the question of whether their frozen features support camera-decoupled motion readout. We probe this question by keeping the video diffusion backbone frozen and training only a lightweight decoder for moving object segmentation (MOS) and motion magnitude estimation (MME). MOS captures the clip-level extent of independently moving objects, while MME estimates the per-pixel, scene-scale-normalized magnitude of world-space 3D displacement between consecutive frames. These readouts are strong enough to outperform dedicated moving object segmentation methods, while comparing favorably with leading feed-forward 4D reconstruction models in 3D motion magnitude estimation. Through controlled comparisons with geometric features, including synthetic-to-real transfer, we provide evidence that this capability is rooted in pretrained video diffusion features rather than decoder training alone. Together, these results establish frozen video diffusion features as effective representations for reading out camera-decoupled scene motion at both object and point levels.

Primary area:
generative models
Keywords:
Video Representation Learning, Moving Object Segmentation, 3D Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u3xdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies whether frozen video diffusion features support motion perception that is decoupled from camera movement. In particular, a decoder on top of a frozen Wan2.2 backbone predicts moving object segmentation (MOS) and the magnitude of world-space 3D displacement (MME). Experiments on FBMS, SegTrack v2 and DAVIS for MOS, and on MOVi-E, Spring and Dynamic Replica for MME, report improvements over existing methods. It further claims that video diffusion features transfer better from synthetic to real videos than DINOv2 and VGGT.

Critical strengths
  • The idea of probing frozen video diffusion features for scene motion that is decoupled from the camera is interesting and well motivated. Separating clip-level moving-object extent from instantaneous 3D motion magnitude is also reasonable.

  • Competitive MOS results are reported without relying on SAM or optical flow, which reduces the dependence on external modules.

Critical weaknesses
  • Lack of controlled experiments to support the central claim. The paper concludes that "this capability is rooted in pretrained video diffusion features rather than decoder training alone" (L024-025). However, no decoder-only or randomly initialized backbone result is reported. It seems that the decoder itself is somewhat complex, with a temporal CNN, a 4-layer ViT and an RGB encoder. The comparison in Tab. 3 is also uneven. DINOv2 and VGGT use a modified temporal decoder (App. B.1) and get no layer sweep, unlike Wan2.2. Besides, it would be better to evaluate a self-supervised video backbone that is not trained for generation, e.g., V-JEPA 2 [ref 1], as a control. I wonder whether the benefit comes from diffusion training or from video pretraining in general. It is highly recommended to add these controls before attributing the ability to video diffusion features.

  • Unconvincing MME comparison with 4D reconstruction models. Under cMAE, predicting zero motion everywhere gives an error of exactly 1.0. Yet MotionCrafter is above 1.0 on all three datasets, and 4RC reaches 1.252 on Spring (Tab. 2), i.e., worse than a static prediction. This suggests problems in how the baseline outputs were converted, or in the metric itself. I suggest reporting trivial baselines, e.g., zero and mask-constant predictions, and a simple pipeline combining depth, camera pose and optical flow. Separate errors on static and dynamic regions would also help. Without these, it is difficult to fully assess the MME advantage over 4D reconstruction models.


  • [ref 1] Assran, Mido, et al. "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning." arXiv:2506.09985 (2025).
Questions for authors
  1. How were layer 30 and σ\sigma=0.25 selected? Fig. 5 sweeps them on FBMS and MOVi-E, which are also reported benchmarks, so a held-out selection split would make the comparison fairer.

  2. It is arguable whether this question "has not been directly examined" (L121-122). MOFT [ref 2] already showed that frozen video diffusion features contain motion-aware channels that can be extracted without training. It might be better to clarify how MOS and MME go beyond it.

  3. What are the decoder size, the peak GPU memory of the 14B backbone, and the variance across training seeds for Tabs. 1-3?


  • [ref 2] Xiao, Zeqi, et al. "Video Diffusion Models are Training-free Motion Interpreter and Controller." NeurIPS 2024.
4 comments

Response to Reviewer u3xd (Comment 1/4)

Thank you for recognizing the motivation and the complementary object-level and point-level readouts, and for identifying concrete controls needed to strengthen the evidence. We address the attribution, evaluation, selection, related-work, and reproducibility concerns below.

Contribution of pretrained features versus decoder training

Q1:

The paper concludes that "this capability is rooted in pretrained video diffusion features rather than decoder training alone" (L024-025). However, no decoder-only or randomly initialized backbone result is reported. It seems that the decoder itself is somewhat complex, with a temporal CNN, a 4-layer ViT and an RGB encoder.

[A1]: Thank you for this suggestion. We retrained the decoder with clean VAE latents or Gaussian noise replacing the DiT features, while retaining the RGB branch and using the same eight-epoch training budget for all three inputs. Both controls substantially underperform pretrained DiT features on MOS and MME (MOS without SAM2).

InputDAVIS J&F ↑FBMS J&F ↑SegTrack J&F ↑MOVi-E cMAE ↓e15 ↓ (%)
Wan2.2 DiT features85.6090.9483.950.39561.3469
Clean VAE latents54.9759.5449.180.61482.5870
Gaussian noise49.0049.1242.681.22434.9978

Fig. 5 and Table 3 provide complementary evidence: performance varies substantially with the feature layer, noise level, and backbone despite comparable decoder training. Together, these results show that decoder capacity alone does not explain the strong MOS and MME results; informative pretrained features are essential to accurate motion readout.

Response to Reviewer u3xd (Comment 2/4)

Decoder comparability and feature-layer selection

Q2a:

DINOv2 and VGGT use a modified temporal decoder (App. B.1) and get no layer sweep, unlike Wan2.2.

[A2a]: Thank you for this helpful suggestion. Our initial use of final-layer features was consistent with SegAnyMo’s choice for DINOv2 and GMOS’s choice for its geometry backbone. We used a single feature layer per backbone to keep feature budgets comparable, accounting for differences in temporal resolution and channel dimensionality.

Regarding the temporal decoder, the adaptation in App. B.1 accommodates the different temporal resolutions of the backbone features. Temporally compressed diffusion features require temporal upsampling, whereas DINOv2 and VGGT provide frame-level features and use a comparable spatiotemporal CNN without temporal upsampling. This design retains temporal modeling while keeping trainable parameter counts similar, as described in the appendix. Overall, these changes only adapt the decoder to the different temporal dimensions of the inputs; they do not change the decoder architecture.

Following your suggestion, we evaluated L11, L17, and L23 for both DINOv2 and VGGT, using a layer-search strategy similar to that used for Wan2.2. These experiments confirm that the final layer is not always optimal. We now use DINOv2 L17 and VGGT L17 global-attention features, keeping the selected layer fixed across tasks and training settings. The updated results for Table 3 are shown below.

BackboneMixed FBMS J&F ↑ (%)Mixed MOVi-E cMAE ↓Synthetic-only FBMS J&F ↑ (%)Synthetic-only MOVi-E cMAE ↓
DINOv2 (L17)86.60.49666.10.467
VGGT (L17, global)83.70.46655.10.395
V-JEPA 2 (L39)88.70.42873.90.381
CogVideoX-5B83.40.46577.50.403
Wan2.1-T2V-1.3B86.90.44180.40.387
Wan2.2-I2V-14B90.90.39686.70.341

We additionally compared global- and frame-attention features at VGGT L17.

VGGT L17 featuresMixed FBMS J&F ↑ (%)Mixed MOVi-E cMAE ↓Synthetic-only FBMS J&F ↑ (%)Synthetic-only MOVi-E cMAE ↓
Global83.700.465855.100.3949
Frame84.760.447942.930.3989

We retain global features as a consistent choice across settings, supported by their stronger cross-domain transfer in this comparison.

Although layer and feature selection improve the baselines, Wan2.2 outperforms all tested DINOv2 and VGGT configurations across the four evaluation settings. We will incorporate these findings into Table 3 and the corresponding discussion in the revised manuscript.

Self-supervised video-pretraining control

Q2b:

Besides, it would be better to evaluate a self-supervised video backbone that is not trained for generation, e.g., V-JEPA 2 [ref 1], as a control. I wonder whether the benefit comes from diffusion training or from video pretraining in general.

[A2b]: Thank you for this suggestion. We have added V-JEPA 2 as a self-supervised video control and used the same independent Kubric validation set as for Wan2.2 to select its feature layer. We evaluated L20, L30, and L39, and report the selected L39 consistently across tasks and training settings in the table above.

We agree that video pretraining is an important source of this capability, and do not claim that it is exclusive to diffusion models. Our motivation for studying video diffusion models is that they are among the most extensively scaled video-pretrained models, while their dense reconstruction objective encourages the preservation of fine-grained, low-level details, such as object boundaries, textures, and small local changes. These properties make them promising representations for motion readout.

Response to Reviewer u3xd (Comment 3/4)

cMAE, trivial baselines, and reconstruction-output conversion

Q3:

Unconvincing MME comparison with 4D reconstruction models. […] This suggests problems in how the baseline outputs were converted, or in the metric itself. I suggest reporting trivial baselines, […] and a simple pipeline combining depth, camera pose and optical flow. Separate errors on static and dynamic regions would also help.

[A3]: Thank you for suggesting these diagnostics. An all-zero prediction indeed has cMAE = 1 on clips with nonzero GT motion. However, predicting zero everywhere misses actual scene motion. The complementary e15 metric in Table 2 measures the percentage of valid pixels whose absolute magnitude error exceeds 15% of the clip's maximum GT motion magnitude. Zero prediction yields e15 values of 5.32%, 3.69%, and 14.26% on MOVi-E, Spring, and Dynamic Replica, respectively, which are worse than those of MotionCrafter, 4RC, and our method on all three datasets.

Following your suggestion, we additionally report zero-motion, all-one, GT mask-constant, RAFT-magnitude, and depth–pose–flow baselines. GT mask-constant predicts one on GT-dynamic pixels and zero elsewhere before scale alignment, providing an oracle test of whether identifying moving regions alone is sufficient. Depth–pose–flow combines VGGT-predicted depth, camera intrinsics and poses with RAFT optical-flow correspondences. It back-projects corresponding pixels into a common world frame and computes their 3D displacement magnitudes, without using GT geometry or motion masks to generate predictions.

To distinguish motion estimation from static-region suppression, we report errors separately on dynamic (nonzero GT displacement) and static (zero GT displacement) pixels. Both regions use the same prediction-to-GT scale fitted on all valid pixels; static error is reported as MAE rather than normalized by zero GT motion.

When evaluated only within dynamic regions, both MotionCrafter and 4RC achieve cMAE below 1 on all three datasets. Their larger static-region errors indicate weaker discrimination between moving and stationary regions. Predicting nonzero motion over large static regions can substantially increase full-image cMAE, while having a smaller effect on e15 when these residual predictions remain below its error threshold. Please see Figure 4 and Appendix Figures 7 (Dynamic Replica) and 8 (DAVIS) for qualitative comparisons illustrating these residual background responses and our more selective motion predictions.

Our method achieves substantially lower static-region errors than both reconstruction baselines across all three datasets, together with lower dynamic-region cMAE in most comparisons. Within dynamic regions, it also outperforms all trivial baselines on all three datasets. Together, these results demonstrate strong local, instantaneous moving/static discrimination and the ability to estimate motion intensity beyond a binary moving/static distinction.

MOVi-E

MethodFull cMAE ↓Full e15 ↓ (%)Dynamic cMAE ↓Static MAE ×1000 ↓
Zero motion1.00005.321.00000.0000
All-one1.74614.691.03374.0259
GT mask-constant — oracle1.32775.281.32770.0000
RAFT magnitude1.75952.650.69674.3342
Depth–pose–flow (VGGT + RAFT)1.04895.480.96440.2841
──────────────────────────────────────────────────
MotionCrafter1.30432.010.57413.1283
4RC0.45341.140.28330.5205
Ours0.35221.110.28090.0933

Spring

MethodFull cMAE ↓Full e15 ↓ (%)Dynamic cMAE ↓Static MAE ×1000 ↓
Zero motion1.00003.691.00000.0000
All-one1.76872.570.89680.6860
GT mask-constant — oracle0.73241.070.73240.0000
RAFT magnitude1.39011.600.72160.3930
Depth–pose–flow (VGGT + RAFT)1.08203.430.97350.0855
──────────────────────────────────────────────────
MotionCrafter1.73051.320.66370.6105
4RC1.25161.800.71360.2675
Ours0.75691.230.66060.0375

Dynamic Replica

MethodFull cMAE ↓Full e15 ↓ (%)Dynamic cMAE ↓Static MAE ×1000 ↓
Zero motion1.000014.261.00000.0000
All-one1.487210.730.81181.5184
GT mask-constant — oracle0.773210.600.77320.0000
RAFT magnitude1.04478.160.60710.8513
Depth–pose–flow (VGGT + RAFT)0.972813.280.90180.1432
──────────────────────────────────────────────────
MotionCrafter1.376711.690.76921.3089
4RC0.58745.360.45470.2052
Ours0.53626.330.52140.0351

Response to Reviewer u3xd (Comment 4/4)

Selection of layer 30 and σ=0.25

Q4:

How were layer 30 and [σ]=0.25 selected? Fig. 5 sweeps them on FBMS and MOVi-E, which are also reported benchmarks, so a held-out selection split would make the comparison fairer.

[A4]: Layer 30 and σ≈0.25 were selected on a held-out subset of our own Kubric-based synthetic dataset, separate from decoder training and the reported benchmarks. The selection set contains 100 scenes, approximately 10% of the size of our original synthetic dataset.

We evaluated all 12 combinations of layers {20, 30, 39} and noise levels σ≈{0.003, 0.25, 0.50, 0.75}, using the same eight-epoch decoder-training budget and selecting the configuration with the lowest cMAE. Layer 30 and σ≈0.25 achieved the best cMAE of 0.3965, followed by layer 39 at the same noise level with 0.4162. Fig. 5 presents benchmark sensitivity analyses; it was not the basis for hyperparameter selection. We will clarify this selection protocol in the revised manuscript.

Noise σ (approx.)L20L30L39
0.0030.50950.45600.4691
0.250.43470.39650.4162
0.500.43530.41640.4367
0.750.51980.46320.4881

Relationship to MOFT and scope of the novelty statement

Q5:

It is arguable whether this question "has not been directly examined" (L121-122). MOFT [ref 2] already showed that frozen video diffusion features contain motion-aware channels that can be extracted without training. It might be better to clarify how MOS and MME go beyond it.

[A5]: Thank you for pointing out this relevant connection. We agree that MOFT has demonstrated motion-aware channels in frozen video diffusion features and their utility for motion transfer. We appreciate your suggestion to clarify how MOS and MME go beyond this finding.

  1. Scene motion independently of camera movement. MOFT studies motion broadly, encompassing both camera and object motion for motion transfer. We specifically evaluate scene motion with camera motion excluded from the target, through MOS at the object and clip level and MME at the local, instantaneous level.
  2. Direct decoding of motion information. MOFT analyzes which internal representations are associated with motion and applies them to motion transfer. We investigate whether motion information in frozen representations can be explicitly decoded into moving-object masks and local motion magnitudes, with accuracy evaluated directly against ground-truth annotations.

We will explicitly discuss MOFT and clarify these distinctions around L121–122.

Decoder size, GPU memory, and seed variance

Q6:

What are the decoder size, the peak GPU memory of the 14B backbone, and the variance across training seeds for Tabs. 1-3?

[A6]: Thank you for requesting these details.

Our decoder contains 74.30M trainable parameters. For the 14B backbone, frozen feature extraction through block 30 requires 26.46 GiB peak allocated GPU memory on a single GPU using BF16 precision, batch size 1, and a 48-frame input at 480 × 848 resolution. This measurement includes the VAE and the retained backbone blocks.

For the main model, we repeated the complete 15-epoch decoder training with three independent training seeds (42, 1, and 2), keeping the backbone frozen and the training and evaluation protocols fixed. The results are reported as mean ± sample standard deviation below.

Dataset / metricRaw modelWith SAM 2 propagation
FBMS J&F ↑92.10 ± 0.2293.00 ± 0.22
SegTrack v2 J&F ↑84.01 ± 0.3286.90 ± 0.39
DAVIS 2017-M J&F ↑87.92 ± 0.2790.71 ± 0.38
MOVi-E cMAE ↓0.355 ± 0.004—
Spring cMAE ↓0.762 ± 0.008—
Dynamic Replica cMAE ↓0.538 ± 0.002—
Reviewer_wAkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies frozen video diffusion features for moving object segmentation (MOS) and motion magnitude estimation (MME). It adds noise to video latents, extracts intermediate features, and trains a decoder for both tasks. The model predicts object motion without explicitly reconstructing the scene or estimating camera motion. The experiments compare different backbones, feature layers, and noise levels.

Critical strengths
  1. Applying video generative features to MOS and MME is interesting. The two tasks cover both complete moving objects and local motion magnitudes. The backbone stays frozen, and only the decoder is trained.

  2. The model transfers well from synthetic training data to real videos for MOS. In Table 3, Wan2.2 trained only on synthetic data achieves 86.7 J&F on FBMS, compared with 54.2 for DINOv2 and 54.6 for VGGT. This suggests that the video features provide useful motion information beyond the decoder’s training domain.

Critical weaknesses
  1. The benefit of adding noise to the latents is not well explained. Figure 5 shows that low but nonzero noise works better than nearly noise-free input. However, the paper offers little discussion of why this happens. The ablation supports the chosen setting, but a clearer explanation would make this finding more informative.

  2. The feature-layer choices may affect the backbone comparison in Table 3. DINOv2 uses its final Transformer block, and VGGT uses its final global-attention block. In contrast, the authors compare several Wan2.2 layers and use an intermediate layer. It is unclear whether the selected layers are suitable for DINOv2 and VGGT on these tasks. Please evaluate a small set of intermediate and late layers for these baselines, or explain the choices. This would help establish whether the performance gap holds under comparable layer selection.

  3. The gains are not strong enough. Many prior works already extract features from pretrained generative models and train a decoder for downstream perception tasks, such as DDPM-based segmentation [1] and VPD [2]. The overall framework therefore offers limited novelty. The main contribution here is applying this approach to MOS and MME, so convincing performance gains on these tasks are particularly important. However, Table 1 shows mixed results. The raw model falls well behind GMOS-S on both DAVIS benchmarks, and the gains with SAM 2 video propagation are often modest. Table 2 reports better cMAE on all three datasets, which is encouraging, but the overall evidence could be stronger. More experiments and analysis are needed to show that transferring video generative features to MOS and MME offers a clear advantage over existing methods.

[1] Label-Efficient Semantic Segmentation with Diffusion Models. ICLR 2022. arXiv:2112.03126.

[2] Unleashing Text-to-Image Diffusion Models for Visual Perception. ICCV 2023. arXiv:2303.02153.

Questions for authors

Please address the questions and concerns raised in Critical Weaknesses. My current assessment is borderline. I would be happy to raise my score if the authors address these concerns.

4 comments

Response to Reviewer wAki (Comment 1/2)

Thank you for highlighting the interest of the two motion readouts and the synthetic-to-real transfer results. We address noise analysis, comparison fairness, contribution, and performance separately below.

Interpretation of the noise ablation

Q1:

The benefit of adding noise to the latents is not well explained. Figure 5 shows that low but nonzero noise works better than nearly noise-free input. However, the paper offers little discussion of why this happens.

[A1]: Thank you for this suggestion. We hypothesize that this reflects the different reconstruction demands at different noise levels. With nearly clean inputs, the DiT input already preserves clear motion cues, so reconstruction may mainly require refining high-frequency appearance details, with less need to infer motion from spatiotemporal context. Moderate noise encourages greater reliance on object structure and spatiotemporal context for recovery, potentially yielding features better suited to motion perception. Excessive noise, however, obscures the input cues needed for accurate motion readout. This provides an intuitive explanation for the intermediate optimum in Fig. 5.

To illustrate this trade-off, we will add representative frames to the appendix and a synchronized video comparison to the supplementary material, showing noisy-latent decoding and one-step recovery at different noise levels. At very low noise levels, the decoded input already retains a clear moving person and motion trajectory, consistent with the hypothesis that the DiT mainly needs to refine high-frequency appearance details. At σ≈0.25, appearance details are perturbed while the overall structure remains recognizable and can be recovered; stronger noise produces more pronounced blurring of the moving person.

Moreover, DIFT [1] also shows that the choice of noise level is task-dependent: it evaluates different timesteps and feature layers and uses different noise levels for semantic, geometric, and temporal correspondence. Our ablation identifies σ≈0.25 as a suitable balance for MOS and MME.

Comparable feature-layer selection

Q2:

Please evaluate a small set of intermediate and late layers for these baselines, or explain the choices. This would help establish whether the performance gap holds under comparable layer selection.

[A2]: Thank you for this helpful suggestion. Our initial use of final-layer features was consistent with SegAnyMo’s choice for DINOv2 and GMOS’s choice for its geometry backbone. We used a single feature layer per backbone to keep feature budgets comparable, accounting for differences in temporal resolution and channel dimensionality.

Following your suggestion, we evaluated L11, L17, and L23 for both DINOv2 and VGGT, using a layer-search strategy similar to that used for Wan2.2. These experiments confirm that the final layer is not always optimal. We now use DINOv2 L17 and VGGT L17 global-attention features, keeping the selected layer fixed across tasks and training settings. The updated results for Table 3 are shown below.

BackboneMixed FBMS J&F ↑ (%)Mixed MOVi-E cMAE ↓Synthetic-only FBMS J&F ↑ (%)Synthetic-only MOVi-E cMAE ↓
DINOv2 (L17)86.60.49666.10.467
VGGT (L17, global)83.70.46655.10.395
V-JEPA 2 (L39)88.70.42873.90.381
CogVideoX-5B83.40.46577.50.403
Wan2.1-T2V-1.3B86.90.44180.40.387
Wan2.2-I2V-14B90.90.39686.70.341

We additionally compared global- and frame-attention features at VGGT L17.

VGGT L17 featuresMixed FBMS J&F ↑ (%)Mixed MOVi-E cMAE ↓Synthetic-only FBMS J&F ↑ (%)Synthetic-only MOVi-E cMAE ↓
Global83.700.465855.100.3949
Frame84.760.447942.930.3989

We retain global features as a consistent choice across settings, supported by their stronger cross-domain transfer in this comparison.

Although layer and feature selection improve the baselines, Wan2.2 outperforms all tested DINOv2 and VGGT configurations across the four evaluation settings. We will incorporate these findings into Table 3 and the corresponding discussion in the revised manuscript.

References

[1] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. "Emergent Correspondence from Image Diffusion." NeurIPS, 2023. https://arxiv.org/abs/2306.03881

Response to Reviewer wAki (Comment 2/2)

Strengths and limitations of the performance evidence

Q3:

The overall framework therefore offers limited novelty. […] However, Table 1 shows mixed results. The raw model falls well behind GMOS-S on both DAVIS benchmarks, and the gains with SAM 2 video propagation are often modest. Table 2 reports better cMAE on all three datasets, which is encouraging, but the overall evidence could be stronger.

[A3]: Thank you for raising this point. We agree that the feature-extraction-and-decoder framework has precedents; however, [2, 3] use image diffusion features for semantic perception, whereas we study whether video generative pretraining encodes scene motion decoupled from camera movement. We summarize several distinctive advantages of our approach over existing methods:

  • Practical annotation and transfer. MME requires camera-decoupled 3D motion labels, which are difficult to obtain at scale, while MOS typically requires a substantial annotation pipeline for identifying independently moving objects and tracing their boundaries with masks. This motivates studying whether large-scale video pretraining can transfer motion perception to domains where real annotations are scarce. Our synthetic-only model retains a substantial advantage over the evaluated image and geometry backbones after the additional layer sweeps in Table 3, and even outperforms several recent methods trained with real data on FBMS (86.7 J&F vs. 81.9 for FlowSAM, 80.7 for SegAnyMo, and 75.5 for GeoMotion).

  • A compact readout without a stack of auxiliary priors. Existing MOS systems often combine geometry, segmentation, or optical-flow priors (GeoMotion with geometry and optical flow, GMOS with geometry and SAM). Our raw model jointly predicts MOS and MME from a single frozen video diffusion backbone, without requiring these additional models.

  • A finer-grained motion target. MOS identifies independently moving objects, whereas MME predicts the world-space 3D displacement magnitude of each visible scene point between consecutive frames. It is therefore a well-defined local and instantaneous motion-intensity target beyond object-level moving/static membership (Figure 4), which is not typically provided by MOS methods.

Comparison with GMOS-S on DAVIS. We acknowledge that our raw model trails GMOS-S on the two DAVIS benchmarks. Table 1 suggests that DAVIS particularly benefits from SAM priors: compared with the SAM-free ABR, SAM-based methods gain only 1.0–3.5 J&F on FBMS and SegTrack v2, but 12.5–19.6 J&F on DAVIS (FlowSAM and GMOS-S). Our raw model uses no SAM prior, yet it outperforms GMOS-S by 7.95 and 2.00 J&F points on FBMS and SegTrack v2, respectively. Consistently, SAM 2 propagation improves our model more on DAVIS (+3.85 and +3.30 J&F on DAVIS 2016-M and 2017-M) than on FBMS (+1.05). Additionally, although the gains from SAM 2 propagation may appear modest in a single run, the multi-seed experiment shows that they are stable: the mean gains are +0.90, +2.89, and +2.79 J&F points on FBMS, SegTrack v2, and DAVIS 2017-M, respectively, with small standard deviations.

Dataset / metricRaw modelWith SAM 2 propagation
FBMS J&F ↑92.10 ± 0.2293.00 ± 0.22
SegTrack v2 J&F ↑84.01 ± 0.3286.90 ± 0.39
DAVIS 2017-M J&F ↑87.92 ± 0.2790.71 ± 0.38
MOVi-E cMAE ↓0.355 ± 0.004—
Spring cMAE ↓0.762 ± 0.008—
Dynamic Replica cMAE ↓0.538 ± 0.002—

[2] Label-Efficient Semantic Segmentation with Diffusion Models. ICLR 2022. arXiv:2112.03126.

[3] Unleashing Text-to-Image Diffusion Models for Visual Perception. ICCV 2023. arXiv:2303.02153.

Thank you for clarifying your concerns.

We agree that prior work has used the feature-extraction-plus-decoder architecture, and we should have stated this more clearly in our previous response. Our aim is not simply to adapt an existing framework to new tasks. Our novel contribution is an insight about video generation models: they implicitly learn to distinguish intrinsic scene motion from camera motion, and this information is already present in their frozen intermediate features. The readout framework is the instrument we use to test it.

Our insight is motivated by the observation that video generators learn to synthesize coherent object motion together with camera movement. Their exposure to large-scale video data, the need to preserve fine-grained motion details, and the requirement to keep objects temporally coherent suggest that their intermediate representations may encode information that separates intrinsic scene dynamics from camera-induced changes. As stated in the Introduction, “Their ability to maintain coherent object structure as objects move across frames suggests that their learned representations may encode both scene motion cues and object structure.” We find that a simple feature-extraction-plus-decoder readout is sufficient to recover this information, and MOS and MME study camera-decoupled motion at two granularities, the object level and the point level. As stated in the Introduction, we investigate “whether frozen video diffusion features can support camera-decoupled motion perception at both object and point levels.” Our results in Tables 1 and 2 demonstrate the strong capabilities of these representations for camera-decoupled motion perception, highlighting the value of this insight. The analysis of Table 3 further shows that this capability originates in the intermediate-layer features we extract, rather than in the decoder. We believe this insight helps us understand and apply the capabilities of video diffusion models for modeling dynamic objects.

Regarding Marigold, it demonstrated the annotation efficiency and transferability of diffusion priors by fine-tuning an image diffusion model for monocular depth estimation. Our study instead addresses a problem that exists only in video, namely separating intrinsic scene motion from camera motion, and it keeps the video backbone entirely frozen. Our results show that the transferable information is already present in the representation itself.

We will revise the manuscript to state our core contribution more clearly.

We thank you again for your question and would welcome further discussion.

Dear Reviewer,

We would like to kindly follow up on our latest response regarding the novelty of our work. We hope it has helped clarify our central contribution.

We would appreciate hearing whether our clarification has addressed your concerns, or if there is anything else we could elaborate on.

Thank you for your time and consideration.

Reviewer_uYEBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether frozen video diffusion features support camera-decoupled motion readout. The authors use a pretrained video diffusion model as a frozen feature extractor and train a lightweight decoder to predict two complementary motion representations: moving object segmentation (MOS), which identifies objects that move independently of the camera, and motion magnitude estimation (MME), which predicts the per-pixel, scene-scale-normalized magnitude of world-space 3D displacement. The method achieves strong performance on several MOS benchmarks and can compare with feed-forward 4D reconstruction methods on MME. The paper further studies different frozen visual backbone features and analyzes the effects of diffusion noise level and transformer layer.

Critical strengths
  1. Provided broad empirical evaluation across the two motion tasks, including multiple datasets and several visual backbones.
  2. Provided meaningful investigation of motion information encoded in video generative representations, with potential value for downstream motion understanding.
Critical weaknesses
  1. It is unclear whether MOS and MME are sufficient to support the broader claim of camera-decoupled motion perception. In particular, MME only keeps motion magnitude and discards direction.
  2. The proposed decoder is trained directly for MME, while baselines it is compared against are optimized for different tasks and are only evaluated through derived motion magnitudes
Questions for authors
  1. Why use MME instead of scene flow? Since MME discards motion direction, it is unclear how much strong MME performance tells us about camera-decoupled motion understanding more broadly.
  2. Why do relatively low but non-zero diffusion noise levels provide better motion readout than nearly noise-free inputs? Figure 5 shows an interesting trend but currently only the observation is reported, without much interpretation.
  3. What properties of video diffusion representations contribute to their advantage over other representations like geometry-oriented features? The gap in Table 3 is substantial, but the paper provides limited discussion of why this gap exists.
5 comments

Response to Reviewer uYEB (Comment 1/2)

Thank you for recognizing the breadth of our evaluation and the potential value of video generative representations for motion understanding. Your concerns help distinguish the scope of the motion readouts from stronger claims about full 3D motion recovery. We address each point below.

Scope of camera-decoupled motion and the choice of MME

Q1:

Why use MME instead of scene flow? Since MME discards motion direction, it is unclear how much strong MME performance tells us about camera-decoupled motion understanding more broadly.

[A1]: We use MME to measure the world-space displacement magnitude of each visible scene point after removing camera motion. It complements MOS at a different granularity: MOS identifies where independently moving objects occur, whereas MME measures local, instantaneous motion intensity, including differences between parts of the same object and across time (similar to instantaneous velocity, Figure 4).

More broadly, this setup allows us to examine whether video generative models capture intrinsic scene dynamics, which may be a particular strength of video-generation pretraining (see our response to Question 4 below). Full scene flow additionally requires recovering 3D motion direction and is more tightly coupled to scene geometry and 3D reconstruction, potentially requiring richer supervision and greater training resources. We therefore use MOS and MME to study camera-decoupled motion extent and intensity perception, without treating magnitude estimation alone as complete 3D motion understanding or reconstruction.

To examine whether the features also support directional information, we additionally replaced scalar MME regression with three-dimensional scene-flow regression and trained a supplementary decoder. We kept Wan2.2 frozen, retained joint MOS supervision, and used the synthetic-only data mixture from Table 3. The results are shown below:

MethodMOVi-E EPE ↓Spring EPE ↓Dynamic Replica EPE ↓
MotionCrafter1.6460.3460.263
4RC0.9260.3660.156
Ours0.7350.2260.167

EPE is multiplied by 100. For each method, we fit one nonnegative scale per clip by minimizing vector EPE, without rotation or translation alignment.

Vector EPE measures both magnitude and direction errors. Our probe achieves lower EPE than both reconstruction baselines on MOVi-E and Spring, and lower EPE than MotionCrafter on Dynamic Replica, where 4RC performs slightly better. These results provide supplementary evidence that the frozen features support motion readout beyond magnitude alone.

Interpretation and fairness of the MME comparison

Q2:

The proposed decoder is trained directly for MME, while baselines it is compared against are optimized for different tasks and are only evaluated through derived motion magnitudes

[A2]: We agree that direct MME supervision is an advantage of our decoder and should be distinguished from the contribution of the pretrained representation. We include 4D reconstruction methods because their predicted scene flow provides a natural estimate of world-space motion magnitude. Table 2 is therefore a comparison of available systems for this readout, demonstrating that useful motion magnitude can be estimated without explicitly reconstructing the full 3D scene; it is not a comparison under identical training objectives.

The controlled representation comparison is provided by Table 3: decoders using the frozen backbones are trained with direct MOS/MME supervision under comparable settings, and we additionally perform layer sweeps for DINOv2 and VGGT. Thus, the differences in this comparison cannot be explained solely by our representation receiving MME supervision. Table 2 establishes the utility of the direct readout, while Table 3 provides the evidence for comparing pretrained features.

Response to Reviewer uYEB (Comment 2/2)

Why does a low but nonzero noise level help?

Q3:

Why do relatively low but non-zero diffusion noise levels provide better motion readout than nearly noise-free inputs? Figure 5 shows an interesting trend but currently only the observation is reported, without much interpretation.

[A3]: Thank you for this suggestion. We hypothesize that this reflects the different reconstruction demands at different noise levels. With nearly clean inputs, the DiT input already preserves clear motion cues, so reconstruction may mainly require refining high-frequency appearance details, with less need to infer motion from spatiotemporal context. Moderate noise encourages greater reliance on object structure and spatiotemporal context for recovery, potentially yielding features better suited to motion perception. Excessive noise, however, obscures the input cues needed for accurate motion readout. This provides an intuitive explanation for the intermediate optimum in Fig. 5.

To illustrate this trade-off, we will add representative frames to the appendix and a synchronized video comparison to the supplementary material, showing noisy-latent decoding and one-step recovery at different noise levels. At very low noise levels, the decoded input already retains a clear moving person and motion trajectory, consistent with the hypothesis that the DiT mainly needs to refine high-frequency appearance details. At σ≈0.25, appearance details are perturbed while the overall structure remains recognizable and can be recovered; stronger noise produces more pronounced blurring of the moving person.

Moreover, DIFT [1] also shows that the choice of noise level is task-dependent: it evaluates different timesteps and feature layers and uses different noise levels for semantic, geometric, and temporal correspondence. Our ablation identifies σ≈0.25 as a suitable balance for MOS and MME.

What explains the advantage over image and geometry features?

Q4:

What properties of video diffusion representations contribute to their advantage over other representations like geometry-oriented features? The gap in Table 3 is substantial, but the paper provides limited discussion of why this gap exists.

[A4]: Thank you for this question. Our study is motivated by three properties of video generative models, which lead us to expect their features to contain transferable information about scene motion independently of camera movement.

  1. Extensive exposure to diverse videos with diverse motion. The demand for high-quality video generation has driven substantial scaling of video data and model capacity, making video generators among the most extensively scaled video foundation models. The Wan technical report describes training on billions of images and videos combined [2], and the official Wan2.2 release reports using 83.2% more videos than Wan2.1 [3]. This exposure provides rich examples of object motion, camera movement, and their combinations across diverse scenes.
  2. Learning fine-grained, low-level details. Generating videos requires more than recognizing high-level semantics: the model must also produce precise object boundaries, textures, and small local changes. Wan learns through dense prediction in video latent space. We expect this objective to encourage features that preserve the fine-grained, low-level information needed to identify moving regions and estimate local, instantaneous motion magnitude.
  3. Learning how scenes evolve over time. DINOv2 processes individual images, while VGGT accepts views without requiring temporal ordering. Video generation explicitly models ordered sequences, requiring the model to learn how objects move, deform, and occlude one another as the camera moves. We expect this joint modeling of object and camera motion to support camera-decoupled motion readout.

Our experiments support the transferability of these features. With synthetic-only training, Wan2.2 achieves 86.7 J&F on FBMS, compared with 66.1 for DINOv2 and 55.1 for VGGT, demonstrating strong transfer of motion readout to real videos.

References

[1] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. "Emergent Correspondence from Image Diffusion." NeurIPS, 2023. https://arxiv.org/abs/2306.03881

[2] Team Wan et al. "Wan: Open and Advanced Large-Scale Video Generative Models." arXiv:2503.20314, 2025. https://arxiv.org/abs/2503.20314

[3] Wan Team. "Wan2.2." Official repository, 2025. See "Complex Motion Generation" for the comparison of training-data sizes. https://github.com/Wan-Video/Wan2.2#wan22

Thank you for your responses to Q1 and Q2. They have addressed my concerns.

However, I still have concerns about the novelty of the model architecture. I understand how your approach differs from existing methods for MOS and MME. But your framework follows the same feature extraction + decoder design used in many previous works. Apart from applying it to new tasks, what are the new contributions?

Simply applying an existing framework to new tasks offers limited novelty. It mainly shows that this framework is robust and transferable. I believe a new paper should provide some new insights beyond this.

Also, the benefits of annotation efficiency and transferability have already been demonstrated in works such as Marigold. Therefore, I do not see this as a new finding either.

I would appreciate further clarification on the novelty of your work.

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback and for recognizing the value of investigating frozen video diffusion features for motion perception. In particular:

  • Reviewer uYEB highlights the broad empirical evaluation across two motion tasks and multiple backbones, and the potential value of studying motion information in video generative representations.
  • Reviewer wAki finds the application to MOS and MME interesting and highlights the strong synthetic-to-real transfer of the frozen video features.
  • Reviewer u3xd finds the camera-decoupled motion readout problem interesting and well motivated, considers the distinction between moving-object extent and instantaneous motion magnitude reasonable, and recognizes the competitive MOS results without SAM or optical flow.

We also recognize a key concern raised by the reviewers: whether the backbone comparison uses fair feature-layer selection. We therefore performed layer sweeps for the baseline backbones. As shown in the updated backbone comparison below, even after tuning these baselines, our Wan2.2 video diffusion features retain a clear advantage in motion readout and synthetic-to-real transfer.

BackboneMixed FBMS J&F ↑ (%)Mixed MOVi-E cMAE ↓Synthetic-only FBMS J&F ↑ (%)Synthetic-only MOVi-E cMAE ↓
DINOv2 (L17)86.60.49666.10.467
VGGT (L17, global)83.70.46655.10.395
V-JEPA 2 (L39)88.70.42873.90.381
CogVideoX-5B83.40.46577.50.403
Wan2.1-T2V-1.3B86.90.44180.40.387
Wan2.2-I2V-14B90.90.39686.70.341

Following the reviewers' suggestions, we have completed additional experiments and analyses, summarized below:

  • An analysis of the possible mechanisms behind the benefit of low but nonzero noise.
  • Updated results for Table 3 with fairer feature-layer selection for the baseline backbones, strengthening the evidence from our synthetic-to-real experiments.
  • Training-seed variability and measured peak GPU memory.
  • Separate errors on dynamic and static regions, together with trivial-baseline comparisons.

We have carefully addressed each reviewer's remaining questions in the corresponding individual responses. We sincerely thank all reviewers and welcome further suggestions to improve our work. We will incorporate these results and analyses into the manuscript as soon as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.