acceptodds
Under review as a conference paper at ICLR 2027

Probing Material Cues in Video Foundation Models

Abstract

General-purpose video foundation models (VidFMs) generate realistic videos that capture how surface appearance evolves across viewpoints and illumination conditions, suggesting that their internal representations may encode intrinsic material information even without explicit material supervision. Motivated by this observation, we investigate the material information embedded in pretrained VidFMs through a lightweight probing framework. We keep the foundation models frozen and measure how well material properties can be recovered from their intermediate representations. Through systematic experiments across synthetic and real-world scenes, together with feature-source ablations, we find that VidFM features provide material cues beyond those available from strong image representations. In controlled settings requiring fine-grained material discrimination, video features show a pronounced advantage over DINOv2 features, while the two representations provide complementary information in more complex scenes. Further analyses show that material recoverability benefits from broader viewpoint coverage and is influenced by object-level semantics. Qualitative synthetic-to-real results further suggest that these material cues remain useful beyond the supervised training domain. Together, these findings provide evidence that large-scale video generative models learn reusable representations containing material-relevant information and characterize the conditions under which these representations offer advantages over image features.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
material estimation, relighting, domain adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5x8cOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to estimate the properties of a digital material model using frozen video models. The idea is to probe the video model intermediate representation to extract the typical channels of a material model, namely albedo, normal, roughness and conductivity. The key message is that large general purpose video models are more efficient than specialized feed forward methods trained from annoted data sets.
The paper give a comprehensive description of the proposed architecture in its main text, completed well by the apprendices. The validation, although often narrowed to "DINO vs video features" is quite complete. I think the main contribution is in the use of frozen video features, optionally fused with DINO feature, to estimate material properties. I think this is an interesting research direction. But I lack elements to really assess how the proposed approach compare to direct specialized methods, both in speed/resolution and quality.

Critical strengths

1/ The paper makes an interesting point in that video features can be very informative for material recognition and estimation. 2/ The fusion of Dino features with the video ones is quite interesting and could certainly be pushed further in the future, although not so critical in the experiment numbers for now.

Critical weaknesses

1/ I think the specialized e.g., feed forward material estimation methods are too quickly discarded. First, this a vast body of work that would deserve a bit more of discussion. Second, although there is certainly interest in knowing how far video models features can go to solve the task, it would still be very informative so see a more comprehensive comparison to specialized models.

2/ While I like the idea of fusing video features, I wonder whether exploiting the training data to adapt Dino feature (fine tuning, LoRA, etc) would help making the them more useful in the fused case.

I believe there are some citation issues (Zeng/Zheng?)

Questions for authors

1/ Would DINOv3 provide any improvement? 2/ Instead of keeping DINO frozen, would an adaptation performed using the same data used to train heads, help improving the DINO side of the feature extraction? 3/ There are other ways to use video models for the task, e.g. VideoNeuMat; how do they compare (at least at a high level) to the proposed method in term of final material appearance (beyond the actual PBR vs neural format)?

Reviewer_fc5NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper investigates whether the frozen internal representations of a pretrained general-purpose video foundation model (VidFM) can be reconstructed to capture PBR material properties (albedo, roughness, metallicity, and normal). The authors extract features from CogVideoX, OpenSora, and WAN, and predict material maps using trainable projectors and probes. Compare with the DINOv2 and material-estimation baselines on the Ball, RoomTour, and DL3DV datasets.
  • The key finding is that WAN-only features recover roughness and metallicity much better than DINOv2-only features in the Ball dataset. In RoomTour, WAN-only is not superior to DINOv2-only across all properties, but combining WAN and DINOv2 yields the best results. The authors argue that this means VidFM representations include material-related cues and can be complementary to image features in complex scenes.
Critical strengths
  • The research question is clear: to directly measure to what extent material-related information is present in pretrained video representations.
  • We distinguish between the results of material estimation and representation analysis, comparing feature sources over a frozen backbone.
  • Distinguishes synthetic ground-truth data from pseudo-label-based real-world data and identifies the numerical limitations of DL3DV.
  • By presenting results for WAN-only, DINOv2-only, and fused features together, we aim to partially distinguish the effects of feature fusion from those of the WAN feature itself.
Critical weaknesses
  • There is no basis for VidFM's standalone feature aside from WAN. The video-only feature-source comparison in Table 1 includes WAN only. Since the main-table results from CogVideoX and OpenSora are feature results fused with DINOv2, it is not possible to conclude that the individual features of each model contain material-relevant information based solely on these results.
  • The effects of temporal pretraining are not dissociated. The backbones compared in the paper differ in architecture, pretraining data, model scale, and learning objective. Therefore, the current experiment only shows differences in performance between pretrained representations; there is no evidence that temporal modeling or video pretraining created those differences.
  • The basis for real-world material accuracy is limited. The figure for DL3DV is the agreement with pseudo-labels, and the authors also state that it is not definitive physical ground truth. The synthetic-to-real evaluation is a qualitative comparison of 10 YouTube videos, and a ground-truth material map is not provided. Therefore, no strong conclusions regarding the accuracy of the actual material or generalization are supported by the current evidence.
Questions for authors
  • Practical significance of the analysis. This paper presents the analytical observation that material information can be recovered from the frozen VidFM feature. However, based on the current results alone, it is unclear what specific benefits this observation would lead to in the design or practical use of the material estimation model. The authors should either clearly demonstrate that this material recoverability analysis provides practical knowledge or design principles beyond simple probing results, or limit the paper's contribution and claims to the scope of analytical observations. This is an important factor in assessing the core contribution and impact of this paper.
  • Can CogVideoX and OpenSora also produce video-only feature results without DINOv2 fusion? If not, are you willing to limit the statement that all VidFMs provide material-relevant features on their own?
Reviewer_SkGuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates recoverability of material properties from frozen video foundation model features. Intermediate diffusion activations are optionally fused with DINOv2 features and decoded by a supervised multiframe probe. Evaluation uses synthetic Ball and RoomTour data with PBR ground truth and DL3DV with pseudo annotations. WAN-only features substantially outperform DINOv2 on Ball roughness and metallicity, while their fusion improves all three material properties on RoomTour; other video backbones are evaluated in the fused setting. Relighting and qualitative synthetic-to-real tests provide downstream checks.

Critical strengths

Broad experimental coverage and complementary validation. The evaluation covers three general-purpose video backbones across Ball, RoomTour, and DL3DV, alongside image-based and material-adapted baselines. This breadth helps characterize how material recoverability varies across backbones and scene settings. Feature-source ablations examine the contribution and complementarity of video and image representations. The downstream evaluation also compares VidFM-MatHead with MVInverse and DiffusionRenderer fine-tuned on the authors’ datasets, providing additional evidence of the full system’s utility for material reconstruction.

Critical weaknesses

Limited novelty beyond an empirical task extension. The core probe closely resembles Huang et al. (2025). The shift from geometric outputs to material maps, together with DINOv2 fusion, appears primarily to adapt this established framework to a new task. The Method section should explicitly distinguish shared design choices from material-specific modifications. The broad comparisons establish useful material-specific empirical findings, particularly recoverability and image–video complementarity. However, the additional analyses offer limited explanation of why the video features help. These findings leave the contribution predominantly empirical, with limited additional methodological or explanatory novelty.

Questions for authors
  • Figure 14 and Table 2 show substantial backbone-dependent differences on RoomTour. What explains why WAN features improve roughness and metallicity estimation over DINOv2, whereas OpenSora and CogVideoX features do not, despite using the same probing architecture and DINOv2 fusion? What material-specific insight can be drawn from these contrasting results?
  • VidFM-MatHead’s predicted material maps appear to exhibit mild temporal flickering in the supplementary video. Have similar fluctuations been observed in the WAN-only probe?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.