acceptodds
Under review as a conference paper at ICLR 2027

Gen4P: Video Generators Are 4D Perceivers

12 views

Abstract

We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. Together, these results demonstrate the potential of video generative pretraining for unified 4D perception. See https://page-a7f39c82.github.io/ for more visual results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feedforward, 3D Tracking, 4D Reconstruction, 4D Perception

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8cQdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gen4P repurposes a pretrained video diffusion model into a feed-forward 4D perception system using a D4RT-style query decoder. It introduces continuous-time query embeddings for variable-length videos and a frozen depth LoRA to strengthen geometric priors. Evaluations across nine datasets demonstrate competitive performance in 3D tracking, depth estimation, and camera pose recovery, particularly in challenging dynamic scenes.

Critical strengths
  1. The method’s design makes sense and is well suited to 4D perception.

  2. The method demonstrates strong empirical performance, supported by both quantitative results and qualitative visualizations.

Critical weaknesses
  1. The central claim is not sufficiently supported by analysis. The title, “Video Generators Are 4D Perceivers,” raises the expectation of a deeper investigation into why video generative pretraining is particularly suitable for 4D perception. As a research paper, I believe it should provide more substantive analysis and controlled comparisons with alternative pretraining approaches to support this claim. Instead, the paper primarily presents a well-engineered adaptation pipeline and demonstrates its effectiveness. Although the overall design and results are compelling, the current evidence mainly demonstrates that the full system works well. It does not fully reveal what makes video generative pretraining uniquely beneficial for 4D perception.
Questions for authors
  1. Can the authors provide further analysis and controlled comparisons with alternative pretraining approaches to clarify why video generative pretraining is particularly well suited to 4D perception?
8 comments
AuthorsOpenReview ↗

W1: Engineering Strengths, Title & Analysis

Summary. We have clarified our title and contribution as adapting a pretrained video generator into a single-step 4D perceiver, supported by added encoder and system comparisons in C1. See below, C1 and C2 for more details.

Engineering Strengths. We thank the reviewer for recognizing the engineering and system contributions of our work.

Title. Our title refers to the demonstrated capability of adapting a pretrained video generator into an effective single-step 4D perceiver under sparse query supervision. Our quantitative and qualitative evaluations support the effectiveness of this adaptation. We have clarified this contribution in C2 and the revised conclusion.

Analysis. We agree that controlled comparisons with alternative pretrained encoders help assess the contribution of the backbone. We have added two comparisons in C1:

  • (r1) replaces the Wan-based encoder with VideoMAEv2 within the Gen4P framework, and
  • (r2) evaluates an adapted OpenD4RT system.

Both use the same data and training settings as (b). The Wan-based variant (b) outperforms both variants on the reported metrics. Please see C1 for the configurations, results, and training curves.

Reminder. The submitted manuscript already includes a noise-level study in Figure 3, which analyzes how input noise affects tracking performance and motivates our clean-latent encoding formulation.

AuthorsOpenReview ↗

Q1: Analysis & Comparisons with Alternative Backbone

Summary. Our added comparisons show faster convergence and better tracking and depth performance for Wan under the same downstream training conditions. We analyze that dynamic video pretraining, a compact scene representation, and avoidance of spatial distortion from square resizing may help explain these advantages, as discussed in C1.

Question Interpretation. The reviewer's question about "why video generative pretraining is particularly well suited" calls for an explanation of its comparative advantages over alternative encoders.
The encoder-replacement comparison in C1 provides empirical support for our backbone choice: Under the same data and training settings, the Wan-based variant achieves faster loss convergence and better tracking and depth performance than the VideoMAEv2-based variant. Please see C1 for the detailed settings and results.

Explanations for Advantages. Video generators are pretrained on large-scale dynamic videos. Learning to generate temporally coherent sequences encourages modeling of cross-frame relationships. We hypothesize that this pretraining encourages internal features rich in motion information, providing potentially useful priors for dynamic 4D perception.

The VAE design enables about twice the temporal compression rate compared to VideoMAEv2, while holding the same spatial compression rate. Thus, given the same encoder frame resolution, the Wan encoder produces nearly half as many scene token, without additional resizing that induces spatial distortion. The shorter scene representation reduces the token-dependent computational burden of cross-attention.

The higher compactness of the representation itself may also reduce redundancy and encourage information aggregation, potentially facilitating downstream 4D readout. Related benefits of compact learned representations have been explored through adaptive token aggregation [1] and the register tokens [2].

Please see C1 for the encoder comparisons at different input resolutions.

[1] TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
[2] VGGT-Ω.

Dear Reviewer 8cQd,

Thank you again for your review. We have added encoder and system comparisons, clarified our core contribution, and expanded our discussion of the potential benefits of video generative pretraining.

With the rebuttal deadline approaching, we would appreciate your feedback on whether these additions address your concerns regarding the supporting evidence and analysis. And please consider updating your score if the concerns have been addressed. Please let us know if any points require further clarification.

Reviewer_WwZJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work repurposes a pretrained video generator (Wan2.1-1.3B) as a feed-forward 4D perceiver. Clean video latent representations undergo a single forward pass through a LoRA-adapted DiT without any denoising steps, and a D4RT-style cross-attention decoder processes point queries related to 3D position, motion, and visibility. Continuous-time query embeddings enable processing of variable-length video clips, and a frozen depth LoRA module derived from DVD is incorporated to inject a geometric prior. Experiments across 3D tracking, video depth estimation, and camera pose estimation tasks achieve state-of-the-art performance on the DR, PStudio, and Sintel benchmarks, alongside ablation studies validating the contributions of its core design choices.

Critical strengths
  1. Our second model variant (Ours-II) outperforms every baseline across multiple dynamic benchmarks that were completely unseen during training: PStudio tracking (APD: 82.11 vs. 79.61), Sintel depth estimation (AbsRel: 0.182 vs. 0.200), and Sintel camera pose estimation. This stands as the most compelling evidence in the paper, as it demonstrates that a video-generator backbone can compete against pipelines built on geometry foundation models in motion-dominant scenarios.

  2. The evaluation is fully transparent: all baselines are scored under a single documented protocol that covers alignment, chunking, and dynamic-point thresholding, and the appendix further reports results for a model variant trained without ADT. This level of transparency makes it possible to verify the reported comparisons independently, including the ADT-related issue discussed below.

Critical weaknesses
  1. The conclusion names leveraging the video-generator prior as "the key to its success", but nothing separates that prior from the 200K-clip training mix.
    • No run swaps only the encoder, e.g., to D4RT's VideoMAE, with decoder, data and schedule fixed.
    • The only same-decoder baseline, the unofficial OpenD4RT checkpoint, is not retrained on this data and trails VGGT on Sintel depth (0.302 vs 0.211), although published D4RT beats VGGT.
  2. Some margins rest on extra supervision that the main tables do not account for.
    • Without ADT training, All APD on ADT falls from 91.93 to 74.69 (Table A3), behind four baselines, while Dynamic APD drops only about 2 points.
    • Ours-II adds no trainable parameters but imports DVD's dense depth training through the frozen LoRA, and DVD itself is absent from Table 2.
Questions for authors
  1. (Rebuttal experiment) With the Table 4 setup, what does the same decoder achieve on the same data with D4RT's VideoMAE encoder in place of Wan? This result would most directly change my rating.
  2. (Clarification) Do any ADT training sequences come from the Apartment scene that all test clips use, and which baselines, or their backbones, saw ADT in training?
  3. (Rebuttal experiment) Please add the video-generator geometry models the paper cites, such as Geo4D and DVD (whose LoRA Ours-II reuses), to Table 2 under the same protocol, to show what Gen4P adds beyond them.
8 comments
AuthorsOpenReview ↗

W1: Separating Vdeo Generator Prior from Training Data Advantage

Summary. We have added VideoMAEv2 comparisons under the same data and training settings. Wan converges faster and performs better on tracking and depth.
Please see C1 and Table RT1 for the settings and results. The VideoMAEv2 variants are listed as (r0) and (r1).

AuthorsOpenReview ↗

W2.1: Result Analysis

Summary. We have expanded the WorldTrack-ADT evaluation and added an All-group decomposition, which shows that ADT supervision mainly benefits in-domain static geometry.

Expanded Group-wise Results. We have expanded ADT results with an additional Static group, shown in Table RT3. Gain is defined as the metric delta along the improved direction.
Table RT3 shows the results on the Static group, with the same level of gains as All group.

All-group Decomposition. We have further analyzed the ADT gains by decomposing the changes in the All-group metrics into static and dynamic contributions, shown in Table RT4.
Specifically, for each clip in the ADT benchmark, we retain each model’s All-group alignment scale and compute metrics for the static and dynamic groups on that scale. Each group’s gain is weighted by its point-count proportion within that clip, followed by equal averaging across clips. Therefore, the two contributions sum to the total All-group improvement for each setting.

As shown in Table RT4, static points account for 67.23% of the total points across all clips, and they contribute 92.22% of the APD gain and almost 100% of the EPE reduction, while the dynamic group contributes rather little or none. These results indicate that ADT supervision primarily benefits the in-domain static geometry.

Evaluation Purposes. We distinguish the purposes of two evaluations:

  • Separate-group metrics (Tables 1 and A3) use group-only alignment to exclude the influence from the other group.
  • The decomposition (Table RT4) uses All-group alignment and weights each group's gain by its point-count proportion to quantify each group’s contribution to the overall improvement.

==============================================

Table RT3: Expanded WorldTrack-ADT. APD Gain = 'Ours-II' −- 'w/o ADT'. EPE Gain = 'w/o ADT' −- 'Ours-II'

SettingAPD ↑\uparrow / AllAPD ↑\uparrow / StaticAPD ↑\uparrow / DynamicEPE ↓\downarrow / AllEPE ↓\downarrow / StaticEPE ↓\downarrow / Dynamic
Ours-II91.9397.6786.830.1120.0410.160
- w/o ADT74.7074.4484.790.2610.2390.148
Gain+17.23+17.23+23.23+23.23+2.04+2.04+0.149+0.149+0.198+0.198−0.012-0.012

==============================================

Table RT4: Decomposition of the gain in WorldTrack-ADT.

GroupPoint Number RatioAPD / Gain (pp)APD / ContributionEPE / Gain (m)EPE / Contribution
Static67.23%+15.89+15.8992.22%+0.149+0.149≈\approx 100.00%
Dynamic32.77%+1.34+1.347.78%≈+10−6\approx +10^{-6}<0.001%<0.001\%
All100%+17.23+17.23100.00%+0.149+0.149100.00%
AuthorsOpenReview ↗

W2.2: Missed Baselines

Summary. We have added Geo4D, GeometryCrafter, and DVD to Table 2.
'Ours-II', which builds upon DVD, outperforms it on Sintel and remains comparable on Bonn and KITTI. Note that DVD is a densely trained video depth model, while Gen4P is a 4D perceiver trained with sparse 4D annotations.

MethodBonn AbsRel ↓Bonn δ<1.25\delta_{<1.25} ↑Sintel AbsRel ↓Sintel δ<1.25\delta_{<1.25} ↑KITTI AbsRel ↓KITTI δ<1.25\delta_{<1.25} ↑
Geo4D0.06097.210.20473.450.08693.83
GeometryCrafter0.05897.310.23967.540.05598.33
DVD0.05397.280.20673.330.05897.73
Ours0.07496.490.18076.130.06296.68
Ours-II0.05997.330.18274.990.05997.32
Reviewer_rsSuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The proposed Gen4P adapts the pretrained Wan2.1-1.3B video diffusion model into a single-step 4D perception model that jointly predicts 3D point tracks, dense geometry, and camera poses from monocular video. Following D4RT's point-query approach, it uses clean video latents, a frozen backbone with trainable 4D-LoRA, and a query decoder to predict point properties across frames. Its main contributions include the continuous-time query, which replaces fixed frame embeddings, and an optional frozen Depth-LoRA that adds depth knowledge. The model is trained on roughly 200K clips from 11 datasets and evaluated on 3D tracking, video depth, and camera pose estimation.

Critical strengths
  1. Combining video generation features with D4RT’s point-query decoder is effective. The noise-level study supports using clean video latents for single-step encoding. Even without Depth-LoRA, the model achieves lower dynamic tracking error on WorldTrack and strong depth results on Sintel. Visual examples also show better preservation of thin structures and less false motion in the background.

  2. The frozen Depth-LoRA adds useful geometry knowledge without increasing trainable parameters. It improves tracking, depth, and camera pose results over the single-LoRA version.

  3. The implementation details make reproduction easier. The paper explains variable-length training with cached video latents, separate resolutions for encoding and local queries, excluded evaluation sequences, and evaluation post-processing.

Critical weaknesses
  1. The benefits of continuous-time queries are unclear. All time embeddings lie on one line, and nearby frames become harder to distinguish as sequence length increases. The decoder also uses relative time, while the encoder uses absolute frame positions. Table 4 shows limited gains, errors increase on longer sequences, and long-video depth and pose evaluation still relies on overlapping chunks.

  2. The extra prediction heads lack clear support. Normals, displacement, 2D positions, and visibility are trained but neither evaluated nor shown to improve the reported tasks. Their value needs separate ablations. The manuscript should also explain how outputs in different camera views remain consistent and which labels and loss masks are available for each training dataset.

Questions for authors
  1. I'm curious how Wan2.1 compares with VideoMAE or VGGT/DA3 encoders when using the same training data, resolution, decoder, and training steps. Also, testing lower LoRA ranks, including 0, 32, 64, and 128, is helpful to show the trade-off between parameter count and accuracy.

  2. Why do time queries use relative positions, while the encoder uses absolute frame positions? Would absolute-time embeddings work better?

  3. How much do the normal, displacement, 2D position, and visibility losses improve tracking and depth accuracy?

  4. Some related work on 4D perception that also predicts dynamic attributes in a feed-forward manner, like motion maps, scene flow, and dense point tracking, is not mentioned in the manuscript, such as [1], [2], and [3].

[1] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos. CVPR 2026.

[2] MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second. CVPR 2026.

[3] DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos. NeurIPS 2025.

8 comments
AuthorsOpenReview ↗

W1: Unclear Function of Continuous Time Embedding

Summary. Our added comparisons show that continuous time embeddings improve ADT accuracy and tracking continuity while supporting temporal flexibility. Longer OOD inputs remain challenging. Depth/pose evaluation uses overlapping chunks for quality and efficiency. The time encodings in encoder and decoder serve different roles and therefore don't need to be correlated. See below and Q2 for more details

Design Motivation. The primary purpose of continuous time embedding is to achieve temporal flexibility, i.e., generalize to variable inference lengths, which we believe is an important feature for 4D perception methods. In Gen4P, we achieve this by replacing the fixed-length lookup table with continuous time embedding.

Such flexibility is critical for 3D tracking, where a multi-chunk handoff strategy fails to work effectively. For discrete embeddings, we have implemented a variant strategy based on OpenD4RT's 'anchor-clip' inference: we fix the first frame and shift the remaining frames to include unseen frames.

Yet it suffers from unrealistic point jittering in OOD length inference, which appears as obvious visual flaws but cannot be fully reflected in metrics:

🔗basketball_3 🔗boxes_6.

Frames 0–48 are inferenced in the native inference pass. From frame 48 to 49, the discrete embeddings will result in an unnegligible sudden shift in point positions. This fundamentally undermines the credibility of 3D tracking and motivates us to adopt the continuous time embedding instead.

Results Clarification. The absolute time description, parameterized as the discrete time embedding ((a) in Table 4), is shown to perform worse on ADT than its continuous counterpart (b). Its comparison directly addresses reviewer's concern about the unclear benefit of continuous embedding. Besides, it enables the critical temporal flexibility, as we explained earlier.

Temporal Distinguishability. In decoding D(q,F)D(q,F), the decoder interprets the time embeddings jointly with the contextualized scene tokens. They jointly support the temporal flexibility.

Reduced spacing between adjacent time embeddings and the need to decode scene tokens at OOD input lengths may jointly contribute to the degradation in Figure 5b. The flatter curve of 4RC indicates lower length sensitivity in the evaluated range, potentially related to its per-frame content conditioning, whereas Gen4P uses content-independent clip-normalized time conditioning.

Gen4P supports inference on sequences of up to 105 frames, with increasing errors at longer OOD lengths. Long-horizon 3D tracking is another valuable and challenging topic in 4D perception, although it's not the main focus of our work and we don't claim our main contribution in that direction. We have included this limitation in the revised manuscript.

Relative & Absolute Encoding. Wan’s 3D RoPE uses clip-local frame indices to parameterize relative positional interactions in encoder self-attention, while the decoder uses clip-normalized times to specify the requested prediction. These encodings serve different roles and don't need to be correlated in cross-attention. See Q2 for more details.

AuthorsOpenReview ↗

W2.1: Auxiliary Losses

Summary. We have added the ablation on the joint auxiliary losses, showing improved video depth estimation and dynamic tracking, with a small All-group EPE trade-off. Please see below and Table RT2 for more details.

To assess the joint effect of auxiliary losses after adaptation to Gen4P, we have trained a variant with all four auxiliary loss weights set to zero, while retaining the 3D position and confidence losses and keeping the remaining settings unchanged.

  • (b) vs. (b-r1): auxiliary supervision improves video depth and dynamic tracking, with a small trade-off in All-group tracking EPEs. This doesn't provide enough motive to discard the auxiliary losses.

Individual auxiliary loss ablations are reported in D4RT (Section 4.4) and we don't claim the auxiliary losses as our contribution. Following D4RT, we adopt these four losses for auxiliary supervision, hoping to increase the information richness of the 4D representation.

Table RT2: Ablation Study on Auxiliary Losses.

ItemAdaptationParams. (B)Tracking / DR / APD ↑\uparrow All / Dyn.Tracking / DR / EPE ↓\downarrow All / Dyn.Tracking / ADT / APD ↑\uparrow All / Dyn.Tracking / ADT / EPE ↓\downarrow All / Dyn.Video Depth / Bonn / AbsRel ↓\downarrowVideo Depth / Bonn / δ<1.25\delta_{<1.25} ↑\uparrowVideo Depth / Sintel / AbsRel ↓\downarrowVideo Depth / Sintel / δ<1.25\delta_{<1.25} ↑\uparrow
(b)ΔW4D512\Delta W_\mathrm{4D}^{512}0.8483.24 / 91.310.186 / 0.09877.54 / 92.190.312 / 0.0770.07497.140.20272.83
(b-r1)- w/o Aux. Losses0.8482.83 / 88.720.184 / 0.11677.76 / 92.060.300 / 0.0810.07996.120.22168.52
AuthorsOpenReview ↗

W2.2: Data Preparation

Summary. We have detailed dataset-specific labels and masks in Appendix B.2. 3D targets use queried camera coordinates. 2D positions and visibility use the observed target view. For constructing tracking annotations, see below and Appendix B.2 for more details.

To briefly explain how the tracking labels are prepared, as recorded in revised Appendix B.2:
The tracking annotations can be read out or rendered from the majority of the datasets. Only for DL3DV, ScanNet++ and MVS-Synth, we backproject the depth maps from several views into 3D and voxelize the merged point clouds. We retain voxels supported by enough source frames and select one representative point per retained voxel. Visibility uses an in-frame proxy with target-depth validity check. The normal annotations can be estimated from depth maps. The masks can be defined based on either 3D or 2D validity.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Acknowledgments, Response Organization & Common Concerns

We appreciate the efforts of the reviewers. Our responses are organized as follows:

  • C1, C2, C3 = Common concerns from more than one reviewer, posted as general comments.
  • W1, W2, ... = Responses to the weaknesses raised by each reviewer.
  • Q1, Q2, ... = Responses to each reviewer’s questions.
  • Tables RT1, RT2, ...= Tables provided in our OpenReview responses.

Replies to individual reviewers are posted under their respective reviews.

AuthorsOpenReview ↗

C1 (1/2): Separating Video Generator Encoder from Data Advantage

Summary. We have added VideoMAEv2 and adapted OpenD4RT comparisons: Gen4P performs better under the same downstream data and training settings. See below and Table RT1 for more details.

To separate the contribution of the pretrained video generator backbone from training data, we have added two ablations:

  • (r0/r1) WVMAEv2-256×{512,256}W_{\text{VMAEv2}}\text{-}256\mathord{\times}\lbrace512,256\rbrace: adapted VideoMAEv2 encoder with the same encoder frames resized to 256×512256\times512 and 256×256256\times256 , replacing the Wan-based encoder in (b). This is to compare with the Wan encoder in Gen4P directly.
  • (r2) OpenD4RT\text{OpenD4RT}: adapted OpenD4RT implementation using 256×256256\times 256 encoder frames. This is to compare the OpenD4RT system with Gen4P on the same datasets. All variants use the same training-data mixture, 45K optimization steps, effective batch size of 16, optimizer, and learning rate schedule.

Configurations. (r0/r1) retain the configuration in Section 4.5 regarding the dataset components, query frame resolution, and decoder depth. They differ only in encoder-frame resolution: 256×512256\times512 for (r0) and 256×256256\times256 for (r1). For more details, please refer to Appendix D in the revised manuscript.

We use the continuous-time embedding with temporal augmentation, which is critical for temporal flexibility. We adjust the set of sampled clip lengths to {20,24,…,48}\{20,24,\ldots,48\} to accommodate VideoMAEv2's temporal patch size. The scene tokens are of dimension 1408, and the query decoder cross-attends to them using 64-dimensional attention heads. The encoder is initialized with pretrained VideoMAEv2-giant weights, with 1.01B and 0.12B trainable parameters in the encoder and decoder, respectively.

(r2) aims to reproduce OpenD4RT while differing in:

  • (r2) resizes the encoder frames to 256×256256\times256, while retaining the query frames at their original resolution, following D4RT. OpenD4RT uses the resized 256×256256\times256 frames as both encoder frames and query frames;
  • Following VideoMAEv2, we rescale encoder frames to [0,1][0,1] and apply mean–std normalization, whereas OpenD4RT applies only rescaling.
  • (r2) uses the same training settings as (r1).

Compared to (r1), (r2):

  • projects the 1408-dimensional VideoMAEv2 encoder features to 1280 dimensions;
  • uses 48 discrete timestep embeddings instead of continuous embeddings;
  • uses the decoder depth of 8 layers, instead of 6; and
  • uses 16 cross-attention heads with 80 dimensions per head and an FFN width of 4480.

Results. We show the extended results in Table RT1, along with the 🔗3d point loss curves.

  • (r0) & (r1) & (r2) vs. (b): (r0), (r1) and (r2) show slower convergence in training than (b) within the same 45k-step budget.
  • (r0) vs (r1): halving the encoder frame resolution to 256×256256\times256 also halves the scene token count, and (r1) outperforms (r0) on all reported tracking and video depth metrics.
  • (r0) vs. (b): at the same encoder frame resolution, (b) outperforms (r0), supporting an advantage of the Wan-based encoder within the Gen4P system.
  • (r0) vs. (d): Wan-based advantage persists under full encoder fine-tuning.
  • (r1) vs. (r2): with the same encoder frame resolution, two variants perform comparably on video depth and DR Tracking, while (r1) is better on ADT tracking.
  • (r2) vs. (b): under the same data and training settings, the Gen4P system outperforms the adapted OpenD4RT system.

Analysis. The compactness of the scene tokens of the Wan encoder may contribute to the encoder advantage. Given the same 256×512256\times 512 inputs, Wan encoder produces 6656 tokens for a 49-frame clip, while VideoMAEv2 encoder produces 12289 tokens for a 48-frame clip, almost twice as many.

Resizing the encoder frame from 256×512256\times512 in (r0) to 256×256256\times256 in (r1) retains the same visual content but halves the scene tokens, improving both tracking and depth estimation. This result suggests that more compact scene tokens may facilitate adaptation, since a shorter K/V sequence reduces the token-dependent computational cost for cross-attention. However, square resizing also induces spatial distortion and may affect encoding quality.

Reproduction Clarification. We clarify that OpenD4RT's training code is mainly configured for fine-tuning its released intermediate checkpoint rather than downstream training initialized from pretrained VideoMAEv2 weights, therefore providing a constrained reproduction reference.

AuthorsOpenReview ↗

C1 (2/2): Separating Video Generator Encoder from Data Advantage

Table RT1: Ablation Study on Different Modules.

ItemAdaptationTrain Params. (B)Tracking / DR / APD ↑\uparrow All / Dyn.Tracking / DR / EPE ↓\downarrow All / Dyn.Tracking / ADT / APD ↑\uparrow All / Dyn.Tracking / ADT / EPE ↓\downarrow All / Dyn.Video Depth / Bonn / AbsRel ↓\downarrowVideo Depth / Bonn / δ<1.25\delta_{<1.25} ↑\uparrow
(r0)\color{purple}{\text{(r0)}}WV-MAEv2-256×512\color{purple}{W_{\text{V-MAEv2}}\text{-}256\mathord{\times}512}1.1359.98 / 65.520.449 / 0.35859.58 / 83.820.608 / 0.1690.24561.83
(r1)\color{purple}{\text{(r1)}}WV-MAEv2-256×256\color{purple}{W_{\text{V-MAEv2}}\text{-}256\mathord{\times}256}1.1367.08 / 68.410.343 / 0.32068.31 / 83.970.428 / 0.1660.12488.44
(r2)\color{purple}{\text{(r2)}}OpenD4RT\color{purple}{\text{OpenD4RT}}1.1760.25 / 70.870.470 / 0.31055.95 / 83.960.654 / 0.1670.13687.23
(a)ΔW4D512†\Delta W_\mathrm{4D}^{512\dagger}0.8483.51 / 91.260.186 / 0.10174.16 / 91.370.358 / 0.1010.07497.13
(r3)\color{purple}{\text{(r3)}}WDiT❄\color{purple}{W_\mathrm{DiT}^{\text{❄}}}0.1441.73 / 49.770.853 / 0.64846.62 / 78.740.920 / 0.2110.23960.63
(r4)\color{purple}{\text{(r4)}}ΔW4D128\color{purple}{\Delta W_\mathrm{4D}^{128}}0.3283.80 / 90.450.173 / 0.10275.03 / 87.820.329 / 0.1220.07795.78
(b)ΔW4D512\Delta W_\mathrm{4D}^{512}0.8483.24 / 91.310.186 / 0.09877.54 / 92.190.312 / 0.0770.07497.14
(c)ΔW4D1024\Delta W_\mathrm{4D}^{1024}1.5482.80 / 91.540.195 / 0.09676.83 / 92.830.325 / 0.0770.07896.96
(d)WDiTW_\mathrm{DiT}1.5381.85 / 90.630.207 / 0.10576.34 / 92.370.328 / 0.0790.08196.62
(e)ΔWDep512\Delta W_\mathrm{Dep}^{512}0.8484.23 / 91.240.172 / 0.09877.92 / 91.790.295 / 0.0820.07197.22
(f)ΔWDep512;❄+ΔW4D512\Delta W_\mathrm{Dep}^{512\\;\text{❄}} + \Delta W_\mathrm{4D}^{512}0.8484.28 / 91.330.168 / 0.09879.99 / 92.900.273 / 0.0750.06997.28

* ΔW{4D, Dep}r\Delta W_{\{\mathrm{4D},\,\mathrm{Dep}\}}^r = Rank-rr {4D, Depth}-LoRA fine-tuning. WDiTW_\mathrm{DiT} = Full DiT fine-tuning. ❄ = Frozen.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.