acceptodds
Under review as a conference paper at ICLR 2027

The Representational Power of Frontier Video Diffusion

Abstract

Do video diffusion models learn useful representations? Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that frontier video diffusion models overcome this limitation. By systematically probing their intermediate activations using zero-shot and parametric probes, we reveal that visual representations evolve across both network depth and noise levels, forming a distributed hierarchical feature space. Importantly, these representations are easily accessible with a single forward pass, and yield state-of-the-art or competitive performance on semantic and geometric understanding tasks without any fine-tuning. Our findings pave the way towards unified backbones for both video generation and perception.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video diffusion, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UPizOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates whether frontier video diffusion models can serve as general-purpose visual encoders, capturing both high-level semantics and low-level geometry. By probing intermediate representations across network depths and noise levels, the authors reveal a hierarchical feature structure and identify optimal extraction points for different tasks. They demonstrate that a single forward pass through a frozen diffusion backbone, combined with lightweight task-specific decoders, achieves competitive or state-of-the-art performance on video understanding tasks.

Critical strengths
  1. The paper challenges previous findings that diffusion models primarily capture low-level geometry but lack high-level semantic understanding. Showing that frontier generative models can support both generation and perception is relevant to building unified visual foundation models;
Critical weaknesses
  1. Missing baselines make the work's claim weak: Previous work[1] is particularly close: it already examines how video diffusion representations vary with layers, noise levels, model size, and training budgets. And some other work[2,3] also conduct similar studies. The authors need to clarify what is the differences compared to these prior work.

[1] Video Diffusion Models Learn the Structure of the Dynamic World [2] Diffusion-Based Action Recognition Generalizes to Untrained Domains [3] GenRec: Unifying Video Generation and Recognition with Diffusion Models

Questions for authors

Please check the major weakness.

3 comments
AuthorsOpenReview ↗

Related work and contribution

Thank you for pointing us to these references (1-DynamicWorld, 2-ActionDiff, 3-GenRec). We added ActionDiff and GenRec in the Related Work section (L090 in blue) and clarify below how our work relates to them.

1-DynamicWorld: Video Diffusion Models Learn the Structure of the Dynamic World (Bao et al.) This is a withdrawn OpenReview submission that is not available on arXiv, so we did not include it in the Related Work. For completeness, we clarify here how it relates to our work. It probes earlier open-source image and video diffusion models (e.g. SD, SVD, Open-Sora) on action recognition (e.g. HMDB51) and dense tasks such as label propagation (DAVIS17). The extraction point is fixed by a heuristic: an early block for classification, a later block for dense tasks, and one default noise level. An ablation covers two blocks and four low noise levels, for SVD only. In contrast, we sweep the full depth × noise range of frontier text-to-video models (Veo3, Wan2.2), and characterise the latent space with zero-shot probes (mutual-kNN alignment with language and vision encoders, shape-texture clustering), and camera motion decodability. This wider sweep leads to a different conclusion. Bao et al. report that the noise level plays a minor role compared to block index. We find that shape bias and alignment with language depend strongly on noise level, peaking at 90% and 60% noise respectively, which is beyond the low-noise range ablated in Bao et al. Finally, we evaluate on more diverse and challenging benchmarks (SSv2 rather than HMDB51, ScanNet depth and segmentation, VATEX captioning), including 3D geometric tasks. When SVD is evaluated frozen under our protocol, it reaches 56.8% on SSv2, compared with 73.5% for Veo3-1P. This result is reported now in our paper (Table 1).

2-ActionDiff (Guimarães et al.) extracts features from a frozen SVD backbone and trains a 6-layer transformer with 8 attention heads on top of them, showing that action recognition generalises to untrained domains (other species, viewpoints and recording contexts). However, a decoder of this capacity can itself learn considerable structure, which makes it difficult to attribute the performance to the diffusion features alone. We instead use a one-block attention readout protocol proposed in Carreira et al. for fair comparison (see general comment). Beyond readout capacity, our study differs in the backbones considered (frontier text-to-video models rather than SVD), in task coverage (geometry, captioning and segmentation in addition to action recognition), and in providing a systematic analysis of the latent space. We have now added SSv2 classification with frozen SVD in Table 1 (in blue), showing that previous generation diffusion models have significantly weaker semantic understanding (56.8% for SVD-1P vs. 73.5% for Veo3-1P).

3-GenRec (Weng et al.) does extensive fine-tuning of the SVD U-Net in two stages to support both generation and recognition. Like ActionDiff, GenRec evaluates action classification only.

W1 - Missing baselines. In conclusion, ActionDiff and GenRec are not directly comparable as they require extensive training. When using their diffusion backbone (SVD) in our frozen evaluation protocol, the results are much weaker, and we are now reporting this explicitly in Table 1 (56.8% for SVD-1P vs. 73.5% for Veo3-1P).

Contribution. To our knowledge, this is the first systematic study of frozen frontier-scale video diffusion models across both semantic and geometric tasks, and it revises the earlier conclusion that diffusion features are weak on semantics.

Reviewer_UPizOpenReview ↗

Thanks to the authors' comments

Thank you to the authors for their response.

Although the authors clarify that the paper is primarily an analysis study, its primary subject area is listed as "Foundation or Frontier Models, including LLMs." Given this positioning, I would expect a more comprehensive and insightful investigation. While the authors claim to be the first to analyze video diffusion models as "general-purpose" visual encoders, substantial prior work has already explored similar ideas using image diffusion models. The paper should better articulate what fundamentally new insights emerge from extending this direction to video diffusion models.

Regarding the claim of a "unified backbone for generation and understanding," I find this statement somewhat overstated, given the extensive existing research on Unified Multimodal Models (UMMs). If the authors claim that "the same backbone can therefore serve both visual generation and understanding," I would expect direct comparisons with representative UMM approaches, such as SenseNova-U1 and NEO, to better contextualize the contribution.

More importantly, to meaningfully strengthen the paper, I encourage the authors to provide deeper insights into the relationship between generation and understanding, rather than simply extending the analysis from image diffusion to video diffusion for video understanding. Following the same logic, similar approaches could potentially be applied to 3D diffusion models for 3D understanding. Merely demonstrating that diffusion features can be repurposed for different understanding tasks does not sufficiently establish a fundamental contribution.

What I believe would be more valuable to the community is a systematic investigation of the underlying synergy between generation and understanding across different modalities. Specifically, why, to what extent, and under what conditions does generative modeling benefit visual understanding? Are there particular tasks, representations, or scenarios where this synergy is especially pronounced? Answering these questions would provide more generalizable insights beyond the specific choice of a video diffusion backbone.

Overall, while I appreciate the authors' clarifications, I believe the current analysis still falls short of providing sufficiently generalizable insights. Therefore, I am tend to maintain my current score.

AuthorsOpenReview ↗

Clarification of scope

Dear Reviewer,

Thank you for the follow-up and for engaging with our response. We made edits in the revised manuscript to clarify the scope, as detailed below:

Unified backbone. We have revised the contribution to "Frozen video generators encode both semantics and geometry", and added UMMs to the Related work section (L109). We emphasise that our goal is to study the understanding capabilities emergent in video generative models.

Image vs video. Video modelling raises questions that image models cannot answer, e.g. motion understanding and temporal dynamics. We added this note in the Related work section (L111) in reference to the above-mentioned UMMs: “These models [UMMs] are mainly evaluated on image semantics and reasoning through language outputs, whereas the representations of motion, temporal dynamics, and dense 3D geometry that emerge from generative training have received little attention. We focus on video diffusion models to study these aspects in detail and provide evidence of strong emergent representations across a diverse set of semantic and geometric tasks.”

Why, to what extent, and under what conditions does generation benefit understanding? We agree that this is the central question. And this is what the analysis and experiments in sections 3 and 4 try to answer. We have now summarised the answers to these questions in Section 5:

  • Noise level: semantic structure is accessible at intermediate-to-high noise levels, where shape bias peaks (90% noise, with purity comparable to DINOv2) and alignment with language peaks (60%), whereas lower-noise features are dominated by appearance. This is consistent with viewing denoising diffusion as approximate autoregression over spatial and temporal frequencies: at high noise, only coarse structure remains, so the model learns global shape, layout, and motion before refining texture. The pattern holds for both Veo3 and the open-weight Wan2.2.

  • Scale: under an identical frozen protocol, SSv2 accuracy increases from 56.8% (SVD) and 59.7% (V-WALT) to 73.5% (Veo3-1P). We report this as a correlation, since more controlled comparisons are needed to establish causality.

  • Task type: semantic information is clearly present in the features. Frozen diffusion features outperform all baselines on motion-centric action recognition (SSv2: 73.5% vs. 72.2% for V-JEPA-H) and are competitive on semantic segmentation (ScanNet: 82.8% vs. 85.1% for DINOv2 and 77.8% for V-JEPA-H), alongside strong results on geometric tasks. The main gap is open-vocabulary captioning, where language-aligned encoders such as SigLIP remain ahead (COCO: 102.0 vs. 118.5 CIDEr), although Veo3-1P leads on SSv2 captioning. We attribute this gap mainly to training: SigLIP is optimised with a contrastive objective on billions of web image--text pairs, which explicitly maps images to a broad open-vocabulary text space, whereas video generators use captions only as conditioning and are never trained to predict or match text.

Reviewer_i9RKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the representational quality of frontier video diffusion models. It extracts intermediate features from a single forward pass on noise-corrupted video latents, using generic text conditioning, and examines their structure across network depth and noise levels. Using alignment metrics and trained readouts, the authors evaluate semantic and geometric capabilities, including action classification, captioning, depth estimation, and camera motion. Veo3 achieves strong frozen-feature results on SSv2 and ScanNet, although captioning remains below the SigLIP baselines. Analyses on Wan2.2 and CogVideoX provide additional evidence from open-weight models.

Critical strengths
  1. Meaningful evidence of semantic and geometric information in frozen generative features. Veo3’s SSv2 and ScanNet results challenge the view that diffusion representations are useful primarily for low-level geometry. Generic conditioning avoids directly supplying target labels or captions to the diffusion backbone.

  2. Complementary and informative probing. The study combines neighborhood alignment, shape-purity measurements, linear and attention probes, and downstream evaluation. The depth/noise sweeps provide useful guidance for feature extraction, while the captioning results transparently reveal remaining weaknesses and include uncertainty estimates across five seeds.

Critical weaknesses
  1. The computational-efficiency claim is unsupported by sufficient measurements. One diffusion forward pass does not establish parity with a discriminative encoder. Veo3’s parameter count is undisclosed, and the reported approximately 7.48B parameters for Wan2.2-1P are not reconciled with the stated Wan2.2-T2V-A14B backbone. The absence of matched latency, memory, and FLOP measurements, including VAE encoding, prevents assessment of the claimed efficiency.

  2. The evidence supports narrower conclusions than some of the paper’s claims. The strongest results depend on proprietary Veo3, whose scale, pretraining data, and potential benchmark overlap are insufficiently characterized. Cross-model comparisons therefore do not isolate generative capability as the cause of stronger representations. Furthermore, the small augmented SSv2 lead lacks uncertainty estimates, and the substantial captioning gaps qualify the claim of broadly overcoming semantic limitations.

Questions for authors
  1. Semantic segmentation would directly test the claim that semantic information is spatially distributed within diffusion features. Could the authors evaluate frozen features using a matched linear or lightweight segmentation readout, comparing against discriminative encoders? This would complement action classification and captioning by assessing spatially localized category semantics.

  2. Can you provide feature-extraction latency, peak memory, and FLOPs on matched hardware and inputs, including VAE encoding and any padding? Does inference terminate at the selected intermediate layer?

  3. Evaluation controls: Were the strongest baselines evaluated with comparable input information, readout capacity, augmentation, and training budgets? Please report classification variability across seeds and clarify how layer/noise selection is separated from final evaluation.

3 comments
AuthorsOpenReview ↗

New experiments, evaluation protocol, and clarifications

Thank you for the detailed and constructive review. Your suggestions on segmentation and evaluation controls have substantially strengthened the paper. We address each point below.

W1 / Q2 – Efficiency. We agree that the claim was overstated. Since compute figures cannot be reported for Veo3, which is proprietary, we have clarified the claim throughout the paper (L043): "features are extracted with a single forward pass, without iterative denoising". This refers only to the number of passes through the model. We make no claim of parity with discriminative encoders in FLOPs, latency or memory.

W1 - Parameter count (Wan2.2) Since Wan2.2-T2V-A14B has 50% depth as the optimal extraction point, and inference terminates at this half-way point, we note its parameter count as 7.26B as the effective number of parameters used. We clarified this at L396.

W2 – Narrower conclusions

  • Reliance on Veo3: The main qualitative findings along the noise axis hold for both Veo3 and Wan2.2: shape bias increases with noise (Figs. 3, 8), alignment with language peaks at t≈60% (Figs. 3, 10). The bimodal depth profile is specific to Veo3.

  • Pre-train - eval contamination: We cannot fully characterise Veo3's training data or rule out overlap with evaluation benchmarks, and the Limitations section now says so (L532).

  • Causality. We agree that more experiments are needed to establish a causal relation between generative objective and representation quality. We have toned down the causal phrasing. The revision only notes that, in our experiments, the stronger generator also yields the stronger representations, and that controlled comparisons are needed to test this further (L334, L521).

  • SSv2 uncertainty, and a correction. While running the suggested multiple-seed experiments, we found that the Veo3-1P SSv2 accuracy in Table 1 of the submission (72.6%) had inadvertently been taken from the 1024-video tuning split rather than from the SSv2 validation set. We have corrected this in the revision. On the validation set, over five runs with different seeds, Veo3-1P reaches 73.5 ± 0.3% (mean ± std; range 73.1–73.8%). All five runs (73.1–73.8%) exceed V-JEPA-H (72.2%).

  • Captioning. We agree that a gap to SigLIP remains on COCO (102.0 vs. 118.5 CIDEr) and VATEX, although Veo3-1P leads on SSv2 captioning. We now state that semantic information is decodable from diffusion features but not yet competitive with language-aligned encoders on open-vocabulary captioning (L060, L527).

Q1 – Semantic segmentation. Thank you for this suggestion. At L400 in the revised draft, we report results on ScanNet segmentation (20 classes, pixel accuracy), compared against DINOv2 and VJEPA2. For Veo3-1P, we used the extraction point (block, noise) that performed best for classification. We show that Veo3-1P results are competitive with DINOv2, while outperforming V-JEPA-H: 82.8% (Veo3-1P) vs. 85.1% (DINOv2) and 77.8% (V-JEPA-2) pixel accuracy. These baselines were run by us under the same readout protocol.

Q3 – Evaluation protocol. Thank you for this comment. We added a dedicated subsection describing the evaluation protocol in subsection 4.1. Readout architecture, capacity, optimiser and training budget follow [4] for all models. Each model is evaluated with its own input format, as documented in App. F: Wan2.2-1P and SVD-1P use 16 frames (stride 2), as in [4], and Veo3-1P uses its native 192-frame input, with clips padded by repeating the last frame. Seed variability is reported above (SSv2 uncertainty), and the separation of extraction-point selection from evaluation is described in the new Sec. 4.1 (see general response).

I appreciate the authors’ clarifications and additional experiments.

W1 / Q2: resolved through narrower efficiency claims and clarified parameter accounting. W2: largely resolved; the conclusions are better qualified, and multi-seed SSv2 results strengthen the evidence. Q1: largely addressed by the new segmentation experiment. Q3: largely resolved by the clarified tuning/evaluation separation and readout protocols.

I find the empirical findings valuable and prefer to maintain my Weak Accept score.

Dear Reviewer, Thank you for your time reviewing our paper and engaging with our response. Your suggestions around semantic segmentation and evaluation controls have strengthened the paper. We appreciate your support.

Reviewer_VxprOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper probes the intermediate representations of video diffusion models (Veo3, Wan2.2, and CogVideoX) across network depths and noise levels, evaluating them on a range of semantic and geometric tasks with frozen backbones.

Critical strengths

The paper evaluates diffusion representations on a broad set of tasks. However, I found it difficult to identify a clear strength beyond this in the current version of the manuscript.

Critical weaknesses
  1. Limited novelty. Analyzing diffusion representations across depths and noise levels for perception tasks has been explored in several prior works, and the paper does not appear to introduce new analysis methods. It would be helpful if the authors could clarify what distinguishes their analysis and findings from existing studies.
  2. Structure and writing flow. The paper would benefit from a clearer narrative connecting its analyses, as the sections currently feel somewhat unevenly organized. In its present form, the manuscript reads more like a collection of probing experiments than a structured study with a clear storyline. Stating the research questions explicitly and organizing the analyses around them could improve readability considerably.
Questions for authors

Could the authors clarify how the optimal (block, noise level) configuration was selected for each downstream task? In particular, was a separate held-out split used for this selection?

4 comments
AuthorsOpenReview ↗

Novelty, structure, evaluation protocol

Thank you for your feedback. We address each point below and refer to the general comment for the list of contributions and the evaluation protocol.

W1 – Novelty. We believe the most relevant prior analyses are [1,2], both discussed in the paper (Sec. 2, L. 093 onwards). Other related works include studies of image diffusion features (already mentioned in the paper) and the video works raised by Reviewer UPiz, which we now discuss in the revised Related Work (L089 in blue). Studies of image diffusion features focus on correspondence and classification tasks and do not address temporal understanding. [1,2] probe frozen video diffusion models at the scale of WALT and conclude that their representations capture low-level geometry but are weak on semantics and poorly aligned with language. ActionDiff (Guimarães et al.) and GenRec (Weng et al.) target action recognition only, and both rely on substantial training: ActionDiff trains a 6-layer transformer with 8 attention heads on top of SVD features, and GenRec trains the SVD U-Net in two stages. In contrast, we study frontier-scale video diffusion models with the generator weights frozen. We read out the features with the lightweight readouts of established protocols [4,5], across six tasks spanning semantics and geometry, and complement this with an in-depth analysis of the latent space.

We do not introduce new probing methods. By using established tools and protocols, our results can be compared directly with the earlier conclusions that diffusion features are weak on semantics. Our findings show that this conclusion does not hold at frontier scale. Under the same frozen protocol, SSv2 accuracy increases from 59.7 (V-WALT) to 73.5 (Veo3-1P), and alignment with text encoders exceeds the best V-WALT configuration. We further find that shape bias increases towards higher noise levels, where it peaks at intermediate depth with a purity comparable to DINOv2, and that Veo3 exhibits a bimodal depth profile. Finally, we show that a single frozen network supports both semantic and geometric readouts, so the same backbone can serve generation and understanding.

W2 – Structure. The revised Introduction now ends with an explicit list of contributions (in blue), pointing to the relevant sections (L047 onwards) to improve readability. In particular, Section 3 addresses what diffusion features of frontier models encode. Section 4 tests how these features transfer to downstream tasks (semantic tasks in section 4.2, geometric tasks in section 4.3). Section 5 discusses the findings and limitations; see general response.

Q1 – Selection of (block, noise). Thank you for raising this. Yes. The extraction point (block, noise) of each diffusion model, and the feature augmentation hyperparameters of Veo3-1P, were selected on a held-out tuning split of 1024 videos from the SSv2 training set, disjoint from all evaluation data. The selected point was then reused without re-selection for all other tasks of the same model (segmentation, depth, and camera pose estimation). Camera motion decodability follows the protocol of [5], which reports the best layer for every model, baselines included. Results for all blocks and noise levels are shown in Figs. 5 and 13, so the sensitivity to this choice is visible. Details are in the new Sec. 4.1 and the general response.

Dear Reviewer,

As the discussion period is coming to an end, we wanted to check whether our response addresses your concerns, in particular on the selection of the extraction points (now described in the new Sec. 4.1), improved structure by explicitly adding a list of contributions at the end of Introduction section, and clarifications regarding existing works added in the Related work section. We would be happy to clarify anything further.

Reviewer_VxprOpenReview ↗

Thank you for your revision

Dear Authors,

Thank you for your revision. I have read it carefully, and I appreciate that you reorganized the structure of the paper so that its contributions are easier to identify.

However, my concern regarding novelty remains unresolved. Prior studies have already analyzed diffusion features in terms of temporal correspondence [1] and semantic alignment in image and video diffusion models [2, 3]. Beyond such analysis papers, there is also a substantial body of work that leverages the geometric and semantic priors of video diffusion features for downstream tasks. While I value the range of experiments presented, much of the analysis appears to reiterate findings that have already been established in these works. I believe the paper would benefit from a more thorough review of the related literature, so that its novelty can be positioned more clearly with respect to prior work. I will maintain my rating (Clear rejection).

[1] Nam et al., Emergent Temporal Correspondences from Video Diffusion Transformers, NeurIPS 2025

[2] Kim et al., Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers, NeurIPS 2025

[3] Jin et al., MATRIX: Mask Track Alignment for Interaction-aware Video Generation, ICLR 2026

Best regards,

Reviewer Vxpr

AuthorsOpenReview ↗

Related works

Dear Reviewer,

Thank you for reading our revision carefully and for acknowledging the improved structure. We appreciate the references and we added them in the Related Work (revised draft at L092, L107, L109). We see them as complementary to our study:

  • [1] (DiffTrack) analyses query–key similarities in the 3D attention of video DiTs to find which layers and timesteps establish temporal correspondence, and applies this to zero-shot point tracking. This is a valuable study, focused on a specific low-level capability and task: correspondence and point tracking. Our work complements this work by analysing more broadly the understanding capabilities emergent in frontier diffusion models, both semantic and geometric, through lightweight readout heads and recent zero-shot probes, instead of attention map analysis. In general, the conclusion of prior work has been that diffusion models have strong capabilities for low-level understanding like point tracking, depth estimation, but they struggle on semantics. We revise this conclusion by showing that at scale, both semantic and geometric knowledge emerges, showing competitive performance with discriminative encoders (DINOv2, V-JEPA, SigLIP) across semantic and geometric tasks: under an identical frozen protocol, SSv2 accuracy rises from 59.7% (V-WALT) and 56.8% (frozen SVD) to 73.5 ± 0.3% (Veo3-1P), ahead of the strongest discriminative baseline (72.2%). On semantic segmentation, we achieve performance similar to DINOv2 and outperform V-JEPA-H (82.8% for Veo3-1P vs. 85.1 for DINOv2 and 77.8% for V-JEPA-H). To our knowledge, this is the first study showing these emergent capabilities in video diffusion models on a wide range of tasks.
  • [2] (Seg4Diff) studies text-to-image MM-DiTs and shows how understanding (semantic segmentation) can improve image generation capabilities. We will discuss this paper in the paragraph with “Most work so far have used understanding to improve generation” where we currently discuss how point tracking has been used to improve generation. This work is similar and focuses on semantic segmentation, having the same outcome of improving generation. Another focus of this work is understanding the impact of text prompting by studying attention maps, similar to DiffTrack. Our work is complementary: we focus on video models, we do not use any text conditioning, and study broad understanding capabilities, semantic and geometric, emergent in frontier diffusion, using readout heads and recent zero-shot probes.
  • [3] (MATRIX) again studies attention maps in diffusion transformers and focuses on improving generation of videos showing interactions by proposing a regularization technique and a new dataset. Similar to Seg4Diff, this work aims to improve video generation. We study the understanding capabilities emergent in frozen video diffusion and show that the simple denoising objective powering diffusion leads to strong representations for a diverse set of tasks, competitive with established visual encoders.

In summary, these works each show that a specific capability emerges in a diffusion model's attention. Our question is different: whether the frozen features of a frontier video generator, read out with lightweight readouts, are competitive with discriminative encoders (DINOv2, V-JEPA, SigLIP) on general understanding tasks, semantic and geometric. The answer revises conclusions of prior works that diffusion does not learn strong semantics. We also report analyses we have not seen for video diffusion models: mutual-kNN alignment with language models, shape-versus-texture bias across noise levels, camera motion decodability with Poincare adapter.

If there are other specific works you have in mind, we would be grateful for the references.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General comment (part 1)

We thank the reviewers for their feedback. We added clarifications and experiments following their input (changes in blue in the revised PDF), which we believe strengthen the paper. Below we clarify the scope, contributions, and evaluation protocol, then summarise the new experiments. We would be grateful if the reviewers could let us know whether these changes address their concerns and we would be happy to provide further clarifications/evidence if needed.

Scope. This is an analysis paper. Recent studies probing frozen video diffusion models [1,2] concluded that diffusion representations capture low-level geometry but struggle with high-level semantics. Our central question is whether this conclusion still holds for frontier-scale video generators. We show that it does not: under the same frozen, lightweight-readout protocol, SSv2 accuracy increases from 59.7% (V-WALT) to 73.5 ± 0.3% (Veo3-1P, mean ± std over five runs). We use established probes so that our results can be compared directly with the earlier negative findings. These are the mutual-kNN alignment metric (proposed in [3] and applied to video encoders in [2]), the 4DS evaluation protocol [4], and the camera motion decodability task of SeeSE3 [5]. Our contribution lies in the findings rather than in new probing tools.

Contributions. The revised Introduction now ends with an explicit list of contributions:

  • Latent space analysis (section 3): We map the evolution of diffusion features across network depth and noise level, using established probes (mutual-kNN alignment, shape-texture clustering, camera motion decodability). Frontier video diffusion models align with language models and with vision encoders such as DINOv2 and VideoMAE, despite having no alignment objective. They show a shape bias at high noise levels comparable to DINOv2, and camera motion decodability on par with geometry-supervised models (DUSt3R, MASt3R). To our knowledge, these properties have not been reported for video diffusion models.

  • Downstream performance (section 4): With lightweight readouts on six tasks (action classification, video captioning, semantic segmentation, depth estimation, camera pose estimation, camera motion decodability), frontier video diffusion features are competitive with strong vision encoders on most tasks, with open-vocabulary captioning as the main exception. This revises the earlier conclusion that diffusion features are weak on semantics.

  • Frozen video generators encode both semantics and geometry (section 5): A video generator trained only with a denoising objective, kept frozen and read out with a single forward pass, supports both semantic and geometric understanding while its generation capabilities remain intact. We discuss when this holds in terms of noise level, model scale, and task type.

Evaluation protocol (now a dedicated subsection 4.1). We follow the evaluation protocols of prior works for fair comparison. The readout architecture and capacity, optimiser, training budget, and data splits for action classification, depth estimation, camera pose estimation, and semantic segmentation are taken from [4]. Camera motion decodability uses the setup from [5]. Baseline results are taken from the original publications, unless stated otherwise [4,5]. Hyperparameters specific to diffusion models, i.e. (block, noise) extraction points for Veo3-1P, Wan2.2-1P and SVD-1P, and the feature augmentation parameters of Veo3-1P, were selected using a tuning split of 1024 videos sampled at random from the SSv2 training set, disjoint from all evaluation data. The optimal selection points identified on this tuning split were then re-used across tasks within the same model. For Veo3-1P and Wan2.2-1P, we report results for all evaluated blocks and noise levels (e.g. Figures 5 and 13), to show the sensitivity to this choice of extraction point. Each model is evaluated with its own input format, as documented in appendix F. For each task, the readout has the same capacity for all models. For captioning, we use a language model fine-tuned on top of the frozen visual diffusion tokens. For all tasks, only the decoders are trained, the diffusion models stay frozen, preserving their generation capabilities.

AuthorsOpenReview ↗

General comment (part 2)

New experiments:

  • SSv2 results for Veo3-1P over five runs: 73.5 ± 0.3%. This also corrects the single-run number in the submission, which had inadvertently been taken from the tuning split instead of the validation set (see reply to Reviewer i9RK).

  • ScanNet semantic segmentation using the same one-block readout protocol; the extraction point (block, noise level) from the classification task was used without re-selection, showing performance competitive to DINOv2 and outperforming VJEPA2: 82.8% (Veo3-1P) vs. 85.1% (DINOv2) and 77.8% (V-JEPA2) pixel accuracy.

  • SSv2 classification using frozen SVD diffusion representations (see reply to Reviewer UPiz), showing substantially lower performance compared to frontier diffusion representations: 56.8% for SVD-1P compared to 73.5% for Veo3-1P.

References:

[1] From Image to Video, Velez et al, https://arxiv.org/pdf/2502.07001

[2] Dynamic reflections, Zhu et al, https://arxiv.org/abs/2511.02767

[3] Platonic representations hypothesis, Huh et al, https://arxiv.org/abs/2405.07987

[4] 4DS: Scaling 4D representations, Carreira et al, https://arxiv.org/abs/2412.15212

[5] SeeSE3: Emergence of 3D Space in Vision Features, Chen et al https://arxiv.org/abs/2607.14228

AuthorsOpenReview ↗

Summary of rebuttal discussion

We thank the reviewers and the AC. Below we summarise the paper's contribution, the changes made during the discussion, and the status of each concern.

Contribution. Recent analyses of frozen video diffusion models [Velez et al.; Zhu et al.] concluded that diffusion features capture low-level geometry but are weak on semantics. Using the same frozen, lightweight-readout protocols of prior works, we show that this conclusion no longer holds at frontier scale:

  • SSv2 action classification: 73.5 ± 0.3% (Veo3-1P, 5 seeds), ahead of strong representation learning baselines V-JEPA-H (72.2%) and far above earlier diffusion models (V-WALT 59.7%, frozen SVD 56.8%).
  • ScanNet semantic segmentation (added in rebuttal): 82.8% Veo3-1P vs. 85.1% DINOv2 and 77.8% V-JEPA-2, using the extraction point picked for classification without re-selection, shows competitive results with the strongest representation learning methods
  • Latent-space findings not previously reported for video diffusion models: strong alignment with language and vision encoders without any alignment objective, shape bias at high noise comparable to DINOv2, and camera-motion decodability on par with geometry-supervised models like DUSt3R/MASt3R. The noise-axis trends hold for both Veo3 and open-weight Wan2.2.
  • Remaining limitation, stated openly: open-vocabulary captioning still trails SigLIP (COCO 102.0 vs. 118.5 CIDEr).

Soundness. No soundness concern remains open. Reviewers i9RK and UPiz both rated soundness 3 ("no substantive concerns"). We added a dedicated evaluation-protocol section (Sec. 4.1): extraction points are chosen on a held-out tuning split that is disjoint from all evaluation data and then reused across tasks; readouts, budgets and splits follow established protocols. While running the multi-seed experiments i9RK asked for, we found and corrected a reporting error (the original 72.6% came from the tuning split). The corrected validation result is 73.5 ± 0.3%, and all five seeds beat the strongest baseline.

Novelty (the main concern of UPiz and Vxpr). Each of the works the reviewers cited differs from ours in essential aspects:

  • ActionDiff, GenRec: action recognition only, with heavy training (a 6-layer transformer on SVD features, or two-stage fine-tuning of SVD). Under our frozen protocol SVD reaches only 56.8%, vs. 73.5% for Veo3-1P (now in Table 1).
  • Bao et al. (withdrawn, not on arXiv): studies older models, using heuristic extraction points, low noise levels only. Our full depth × noise sweep reaches a different conclusion: noise level strongly impacts semantics, which emerge at higher noise levels compared to their study.
  • DiffTrack, Seg4Diff, MATRIX: analyse attention maps for one capability (point correspondence leveraged for point tracking) or use understanding to improve generation.

We thank the reviewers for these references and have added all suggested published works to the Related Work section (except for Bao et al.). These works study different questions (single-task action recognition with heavy training, attention-map analysis of a single capability, or using understanding to improve generation), so they complement our analysis rather than overlap with it. None evaluates frozen frontier video-generator features against discriminative encoders across both semantic and geometric tasks, or revises the earlier conclusion that video diffusion features are weak on semantics.

Other changes made in response to the reviews.

  • An explicit list of contributions (Vxpr appreciated the change).
  • Clarified claims: efficiency now means "a single forward pass, without iterative denoising"; causal wording removed; UMMs added to Related Work.
  • Extended the Conclusion section (Section 5) to reiterate the findings, answering UPiz's follow-up question, and we detailed the limitations to acknowledge risks of benchmark overlap that are beyond our control.

We believe the revised paper brings strong empirical evidence for the extensive understanding capabilities emerging in frontier diffusion models that has not been previously reported. We would be grateful if the AC took these revisions into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.