- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper investigates whether frontier video diffusion models can serve as general-purpose visual encoders, capturing both high-level semantics and low-level geometry. By probing intermediate representations across network depths and noise levels, the authors reveal a hierarchical feature structure and identify optimal extraction points for different tasks. They demonstrate that a single forward pass through a frozen diffusion backbone, combined with lightweight task-specific decoders, achieves competitive or state-of-the-art performance on video understanding tasks.
Critical strengths
- The paper challenges previous findings that diffusion models primarily capture low-level geometry but lack high-level semantic understanding. Showing that frontier generative models can support both generation and perception is relevant to building unified visual foundation models;
Critical weaknesses
- Missing baselines make the work's claim weak: Previous work[1] is particularly close: it already examines how video diffusion representations vary with layers, noise levels, model size, and training budgets. And some other work[2,3] also conduct similar studies. The authors need to clarify what is the differences compared to these prior work.
[1] Video Diffusion Models Learn the Structure of the Dynamic World [2] Diffusion-Based Action Recognition Generalizes to Untrained Domains [3] GenRec: Unifying Video Generation and Recognition with Diffusion Models
Questions for authors
Please check the major weakness.
3 comments
Thanks to the authors' comments
Thank you to the authors for their response.
Although the authors clarify that the paper is primarily an analysis study, its primary subject area is listed as "Foundation or Frontier Models, including LLMs." Given this positioning, I would expect a more comprehensive and insightful investigation. While the authors claim to be the first to analyze video diffusion models as "general-purpose" visual encoders, substantial prior work has already explored similar ideas using image diffusion models. The paper should better articulate what fundamentally new insights emerge from extending this direction to video diffusion models.
Regarding the claim of a "unified backbone for generation and understanding," I find this statement somewhat overstated, given the extensive existing research on Unified Multimodal Models (UMMs). If the authors claim that "the same backbone can therefore serve both visual generation and understanding," I would expect direct comparisons with representative UMM approaches, such as SenseNova-U1 and NEO, to better contextualize the contribution.
More importantly, to meaningfully strengthen the paper, I encourage the authors to provide deeper insights into the relationship between generation and understanding, rather than simply extending the analysis from image diffusion to video diffusion for video understanding. Following the same logic, similar approaches could potentially be applied to 3D diffusion models for 3D understanding. Merely demonstrating that diffusion features can be repurposed for different understanding tasks does not sufficiently establish a fundamental contribution.
What I believe would be more valuable to the community is a systematic investigation of the underlying synergy between generation and understanding across different modalities. Specifically, why, to what extent, and under what conditions does generative modeling benefit visual understanding? Are there particular tasks, representations, or scenarios where this synergy is especially pronounced? Answering these questions would provide more generalizable insights beyond the specific choice of a video diffusion backbone.
Overall, while I appreciate the authors' clarifications, I believe the current analysis still falls short of providing sufficiently generalizable insights. Therefore, I am tend to maintain my current score.
Clarification of scope
Dear Reviewer,
Thank you for the follow-up and for engaging with our response. We made edits in the revised manuscript to clarify the scope, as detailed below:
Unified backbone. We have revised the contribution to "Frozen video generators encode both semantics and geometry", and added UMMs to the Related work section (L109). We emphasise that our goal is to study the understanding capabilities emergent in video generative models.
Image vs video. Video modelling raises questions that image models cannot answer, e.g. motion understanding and temporal dynamics. We added this note in the Related work section (L111) in reference to the above-mentioned UMMs: “These models [UMMs] are mainly evaluated on image semantics and reasoning through language outputs, whereas the representations of motion, temporal dynamics, and dense 3D geometry that emerge from generative training have received little attention. We focus on video diffusion models to study these aspects in detail and provide evidence of strong emergent representations across a diverse set of semantic and geometric tasks.”
Why, to what extent, and under what conditions does generation benefit understanding? We agree that this is the central question. And this is what the analysis and experiments in sections 3 and 4 try to answer. We have now summarised the answers to these questions in Section 5:
-
Noise level: semantic structure is accessible at intermediate-to-high noise levels, where shape bias peaks (90% noise, with purity comparable to DINOv2) and alignment with language peaks (60%), whereas lower-noise features are dominated by appearance. This is consistent with viewing denoising diffusion as approximate autoregression over spatial and temporal frequencies: at high noise, only coarse structure remains, so the model learns global shape, layout, and motion before refining texture. The pattern holds for both Veo3 and the open-weight Wan2.2.
-
Scale: under an identical frozen protocol, SSv2 accuracy increases from 56.8% (SVD) and 59.7% (V-WALT) to 73.5% (Veo3-1P). We report this as a correlation, since more controlled comparisons are needed to establish causality.
-
Task type: semantic information is clearly present in the features. Frozen diffusion features outperform all baselines on motion-centric action recognition (SSv2: 73.5% vs. 72.2% for V-JEPA-H) and are competitive on semantic segmentation (ScanNet: 82.8% vs. 85.1% for DINOv2 and 77.8% for V-JEPA-H), alongside strong results on geometric tasks. The main gap is open-vocabulary captioning, where language-aligned encoders such as SigLIP remain ahead (COCO: 102.0 vs. 118.5 CIDEr), although Veo3-1P leads on SSv2 captioning. We attribute this gap mainly to training: SigLIP is optimised with a contrastive objective on billions of web image--text pairs, which explicitly maps images to a broad open-vocabulary text space, whereas video generators use captions only as conditioning and are never trained to predict or match text.
Related work and contribution
Thank you for pointing us to these references (1-DynamicWorld, 2-ActionDiff, 3-GenRec). We added ActionDiff and GenRec in the Related Work section (L090 in blue) and clarify below how our work relates to them.
1-DynamicWorld: Video Diffusion Models Learn the Structure of the Dynamic World (Bao et al.) This is a withdrawn OpenReview submission that is not available on arXiv, so we did not include it in the Related Work. For completeness, we clarify here how it relates to our work. It probes earlier open-source image and video diffusion models (e.g. SD, SVD, Open-Sora) on action recognition (e.g. HMDB51) and dense tasks such as label propagation (DAVIS17). The extraction point is fixed by a heuristic: an early block for classification, a later block for dense tasks, and one default noise level. An ablation covers two blocks and four low noise levels, for SVD only. In contrast, we sweep the full depth × noise range of frontier text-to-video models (Veo3, Wan2.2), and characterise the latent space with zero-shot probes (mutual-kNN alignment with language and vision encoders, shape-texture clustering), and camera motion decodability. This wider sweep leads to a different conclusion. Bao et al. report that the noise level plays a minor role compared to block index. We find that shape bias and alignment with language depend strongly on noise level, peaking at 90% and 60% noise respectively, which is beyond the low-noise range ablated in Bao et al. Finally, we evaluate on more diverse and challenging benchmarks (SSv2 rather than HMDB51, ScanNet depth and segmentation, VATEX captioning), including 3D geometric tasks. When SVD is evaluated frozen under our protocol, it reaches 56.8% on SSv2, compared with 73.5% for Veo3-1P. This result is reported now in our paper (Table 1).
2-ActionDiff (Guimarães et al.) extracts features from a frozen SVD backbone and trains a 6-layer transformer with 8 attention heads on top of them, showing that action recognition generalises to untrained domains (other species, viewpoints and recording contexts). However, a decoder of this capacity can itself learn considerable structure, which makes it difficult to attribute the performance to the diffusion features alone. We instead use a one-block attention readout protocol proposed in Carreira et al. for fair comparison (see general comment). Beyond readout capacity, our study differs in the backbones considered (frontier text-to-video models rather than SVD), in task coverage (geometry, captioning and segmentation in addition to action recognition), and in providing a systematic analysis of the latent space. We have now added SSv2 classification with frozen SVD in Table 1 (in blue), showing that previous generation diffusion models have significantly weaker semantic understanding (56.8% for SVD-1P vs. 73.5% for Veo3-1P).
3-GenRec (Weng et al.) does extensive fine-tuning of the SVD U-Net in two stages to support both generation and recognition. Like ActionDiff, GenRec evaluates action classification only.
W1 - Missing baselines. In conclusion, ActionDiff and GenRec are not directly comparable as they require extensive training. When using their diffusion backbone (SVD) in our frozen evaluation protocol, the results are much weaker, and we are now reporting this explicitly in Table 1 (56.8% for SVD-1P vs. 73.5% for Veo3-1P).
Contribution. To our knowledge, this is the first systematic study of frozen frontier-scale video diffusion models across both semantic and geometric tasks, and it revises the earlier conclusion that diffusion features are weak on semantics.