- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Gen4P repurposes a pretrained video diffusion model into a feed-forward 4D perception system using a D4RT-style query decoder. It introduces continuous-time query embeddings for variable-length videos and a frozen depth LoRA to strengthen geometric priors. Evaluations across nine datasets demonstrate competitive performance in 3D tracking, depth estimation, and camera pose recovery, particularly in challenging dynamic scenes.
Critical strengths
-
The method’s design makes sense and is well suited to 4D perception.
-
The method demonstrates strong empirical performance, supported by both quantitative results and qualitative visualizations.
Critical weaknesses
- The central claim is not sufficiently supported by analysis. The title, “Video Generators Are 4D Perceivers,” raises the expectation of a deeper investigation into why video generative pretraining is particularly suitable for 4D perception. As a research paper, I believe it should provide more substantive analysis and controlled comparisons with alternative pretraining approaches to support this claim. Instead, the paper primarily presents a well-engineered adaptation pipeline and demonstrates its effectiveness. Although the overall design and results are compelling, the current evidence mainly demonstrates that the full system works well. It does not fully reveal what makes video generative pretraining uniquely beneficial for 4D perception.
Questions for authors
- Can the authors provide further analysis and controlled comparisons with alternative pretraining approaches to clarify why video generative pretraining is particularly well suited to 4D perception?
8 comments
Q1: Analysis & Comparisons with Alternative Backbone
Summary. Our added comparisons show faster convergence and better tracking and depth performance for Wan under the same downstream training conditions. We analyze that dynamic video pretraining, a compact scene representation, and avoidance of spatial distortion from square resizing may help explain these advantages, as discussed in C1.
Question Interpretation.
The reviewer's question about "why video generative pretraining is particularly well suited" calls for an explanation of its comparative advantages over alternative encoders.
The encoder-replacement comparison in C1 provides empirical support for our backbone choice:
Under the same data and training settings, the Wan-based variant achieves faster loss convergence and better tracking and depth performance
than the VideoMAEv2-based variant. Please see C1 for the detailed settings and results.
Explanations for Advantages. Video generators are pretrained on large-scale dynamic videos. Learning to generate temporally coherent sequences encourages modeling of cross-frame relationships. We hypothesize that this pretraining encourages internal features rich in motion information, providing potentially useful priors for dynamic 4D perception.
The VAE design enables about twice the temporal compression rate compared to VideoMAEv2, while holding the same spatial compression rate. Thus, given the same encoder frame resolution, the Wan encoder produces nearly half as many scene token, without additional resizing that induces spatial distortion. The shorter scene representation reduces the token-dependent computational burden of cross-attention.
The higher compactness of the representation itself may also reduce redundancy and encourage information aggregation, potentially facilitating downstream 4D readout. Related benefits of compact learned representations have been explored through adaptive token aggregation [1] and the register tokens [2].
Please see C1 for the encoder comparisons at different input resolutions.
[1] TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
[2] VGGT-Ω.
Dear Reviewer 8cQd,
Thank you again for your review. We have added encoder and system comparisons, clarified our core contribution, and expanded our discussion of the potential benefits of video generative pretraining.
With the rebuttal deadline approaching, we would appreciate your feedback on whether these additions address your concerns regarding the supporting evidence and analysis. And please consider updating your score if the concerns have been addressed. Please let us know if any points require further clarification.
W1: Engineering Strengths, Title & Analysis
Summary. We have clarified our title and contribution as adapting a pretrained video generator into a single-step 4D perceiver, supported by added encoder and system comparisons in C1. See below, C1 and C2 for more details.
Engineering Strengths. We thank the reviewer for recognizing the engineering and system contributions of our work.
Title. Our title refers to the demonstrated capability of adapting a pretrained video generator into an effective single-step 4D perceiver under sparse query supervision. Our quantitative and qualitative evaluations support the effectiveness of this adaptation. We have clarified this contribution in C2 and the revised conclusion.
Analysis. We agree that controlled comparisons with alternative pretrained encoders help assess the contribution of the backbone. We have added two comparisons in C1:
Both use the same data and training settings as (b). The Wan-based variant (b) outperforms both variants on the reported metrics. Please see C1 for the configurations, results, and training curves.
Reminder. The submitted manuscript already includes a noise-level study in Figure 3, which analyzes how input noise affects tracking performance and motivates our clean-latent encoding formulation.