- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is motivated by the observation that recent video generation models often fail to generate videos that faithfully preserve realistic physical properties. It proposes a benchmark to evaluate and analyze whether the representation learning methods on which video generation models (e.g., RAG) rely faithfully encode information related to physical properties. To this end, the benchmark introduces a dataset covering a wide range of physical property families, from fluid, solid mechanics, and dynamics to optics. Based on this dataset, the benchmark evaluates how well the representation learning methods capture physical properties through three tasks: text-video retrieval, physical property regression, and pair classification.
Critical strengths
According to the paper, unlike existing similar benchmarks, the proposed benchmark enables the evaluation and analysis of a diverse range of physical properties, from fluid, solid mechanics, and dynamics to optics, while also considering physical quantities. This allows for a more fine-grained evaluation of physical properties beyond simply assessing scene semantics. The tree evaluation tasks seem to further enable the physical understanding of representation learning methods to be evaluated and analyzed from multiple aspects.
Critical weaknesses
CW1. Regarding the classification task utilized in the paper, could the classification performance of a model (for both within- and cross-family tasks) be achieved through an easy answer that does not require model to understand physical properties, but instead relies on simple visual difference? For example, within-family classification may be possible simply by recognizing difference in objects in the solid mechanics family. Moreover, cross-family classification may be even easier, as the classes could potentially be distinguished based on visually apparent differences in image statistics, which could explain the higher classification accuracy compared to within-family classification (in Table 2). If this is the case, does high classification accuracy necessarily indicate that a model has a better understanding of physical properties, or could it instead reflect the model’s ability to recognize superficial visual differences between the classes?
CW2. Regarding the retrieval task, could the near-chance-level performance of most models be attributed not to a lack of understanding of physical properties, but simply to poor generalization to out-of-distribution data? For example, most models are trained primarily on natural videos. Therefore, their poor performance on the simulated data in this benchmark, which differs substantially from model’s training data, could instead reflect limited robustness to out-of-distribution data rather than an inability to understand physical properties.
Questions for authors
As pointed out in the critical weaknesses above, I believe that two of three tasks proposed by the paper for evaluating models’ physical knowledge, namely retrieval and classification, may be confounded by factors unrelated to physical knowledge. Therefore, I would like to know whether the benchmark provides any analysis or interpretation that can account for these factors and help disentangle their effects from the models’ actual understanding of physical properties.
4 comments
(2/2) Response to Reviewer EKUt
3. Disentangling confounding factors across tasks
Thanks for this comment. A central motivation of our benchmark is to evaluate physical representations from multiple complementary perspectives, rather than using a single task as a definitive measure of physical understanding and reasoning.
As described in Section 2.2, the three tasks assess different capabilities:
Classification: Cross-family classification primarily measures coarse-grained discrimination between physical systems, where visual differences may contribute to performance. Within-family classification reduces this confounding factor by comparing instances of the same physical system with different conditions and responses, providing a more fine-grained evaluation.
Retrieval: Retrieval evaluates the alignment between detailed physical descriptions and video representations. While it provides a more challenging measure of fine-grained cross-modal correspondence, its performance may also be affected by visual differences and domain shift.
Regression: Regression provides a more direct quantitative evaluation of physical-property recoverability. We use controlled simulations in which the target physical property varies while other relevant conditions remain fixed, allowing us to examine whether the corresponding information can be recovered from frozen video representations.
Importantly, the discrepancies across these tasks are themselves informative. Pretrained models can achieve relatively strong quantitative regression performance while remaining close to chance on within-family classification and performing poorly on retrieval. This suggests that quantitative information recoverability, fine-grained discrimination, and cross-modal alignment are distinct capabilities that should not be conflated. The complementary evaluations help characterize these differences and reveal limitations that would be obscured by a single task.
Thank you for the response. The authors acknowledge that visual differences may contribute to cross-family classification and that domain shift may contribute to the low retrieval performance. However, no additional analysis is provided to separate these factors from physical understanding. In particular, within-family cases can still differ in visible properties such as object geometry, and both the regression probes and physics adaptation rely on simulated data, leaving the effect of domain shift unexamined. For these reasons, I will maintain my score.
Thank you to the authors for the detailed rebuttal and additional experiments, which addressed several of my concerns.
However, my primary concern remains. While the pixel-based regression results provide useful evidence of recoverability under controlled conditions, they do not fully establish that individual video-text pairs are uniquely distinguishable, especially when different physical configurations producing similar observations.
I appreciate the additional clarifications and experiments, but I believe this fundamental concern remains insufficiently resolved. Therefore, I will maintain my current rating.
(1/2) Response to Reviewer EKUt
Thank you for the careful evaluation of our work and the insightful questions. We appreciate the opportunity to clarify the scope of our benchmark and the complementary roles of the three evaluation tasks. We address your concerns point-by-point as follows:
1. Superficial Visual Differences in Pair Classification
Thanks for the comment. We agree that visual cues may contribute to cross-family evaluation, and would like to clarify that within-family and cross-family classification are designed to evaluate different levels of physical discrimination.
Cross-family classification evaluates discrimination across different physical systems. We acknowledge that coarse visual and semantic differences between families may contribute to performance in this setting. Therefore, high cross-family accuracy alone should not be interpreted as evidence of fine-grained physical understanding.
Within-family classification, on the other hand, uses negative examples from the same simulation family. The positive and negative cases share the same underlying physical system and broad scene characteristics, but differ in their case-specific conditions and physical responses. This design reduces reliance on simple scene-category recognition and provides a more challenging evaluation of fine-grained video-text correspondence. While visual cues may still contribute, this setting is less susceptible to the coarse visual differences present in cross-family classification.
Importantly, the substantial performance gap between these two settings is itself informative. As shown in Table 2, pretrained embedding models achieve relatively strong cross-family classification but remain close to chance on within-family classification. This suggests that existing embeddings are substantially better at distinguishing broad physical scenarios than at aligning fine-grained physical distinctions.
We also emphasize that pair classification is only one of three complementary evaluation tasks. Our retrieval evaluation requires matching detailed physical descriptions against larger video candidate pools (700-2700 videos depending on the specific physics branch), while physical-property regression directly measures the recoverability of continuous physical quantities from video embeddings. These tasks provide additional perspectives beyond categorical visual discrimination.
2. Retrieval and Out-of-distribution Generalization
We agree that the distribution gap between natural and simulated videos may contribute to the low zero-shot retrieval performance. However, our findings suggest that the issue is not simply an inability to extract meaningful physical information from simulated videos.
First, as shown in our regression evaluation, pretrained video representations already support quantitative physical-property prediction through lightweight linear probes. In particular, frozen LCO-Embedding-7B representations achieve an average nRMSE of 0.0202 across the ten regression families. This indicates that meaningful physical information is recoverable from simulated videos, even without physics-specific representation training.
Second, physics-aware contrastive adaptation substantially improves retrieval performance on the same benchmark evaluation distribution. For example, LCO-Embedding-7B improves Fluid R@10 from 4.9% to 17.9% and Dynamics R@10 from 5.7% to 21.0% after physics adaptation. These results demonstrate that cross-modal alignment with physical descriptions can be substantially improved without changing the evaluation distribution. The improvements would have been more difficult if the knowledge is not encoded in the base foundation models, according to studies on the relationship between base model capabilities and their representation quality after contrastive learning.
These findings suggest that physical information can be encoded in pretrained representations without being effectively exposed through embedding similarity. We acknowledge that domain shift may contribute to the observed retrieval difficulties, and that physics-specific adaptation may also improve familiarity with the simulation domain. Therefore, we do not interpret low zero-shot retrieval performance as definitive evidence that pretrained models lack physical knowledge. Instead, our benchmark aims to characterize how effectively current representations capture and expose fine-grained physical information under a controlled evaluation setting.