acceptodds
Under review as a conference paper at ICLR 2027

World Embedding Benchmark

Abstract

Physical fidelity has received increasing attention in world models and video generation, yet how video representations encode physical information remains less understood. We introduce the World Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification. We use these tasks to distinguish cross-modal physical alignment from the recoverability of quantitative physical information. Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings. Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability. Finally, we use the embeddings to retrieve reference videos for retrieval-augmented generation with MiniMax-H3. Retrieved references improve the physical fidelity of generated videos, with stronger retrieval models yielding larger gains in our experiments. Together, these findings highlight the need to evaluate physical alignment and property recoverability jointly, and demonstrate the utility of physical representations for improving video generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, embedding, physics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EKUtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is motivated by the observation that recent video generation models often fail to generate videos that faithfully preserve realistic physical properties. It proposes a benchmark to evaluate and analyze whether the representation learning methods on which video generation models (e.g., RAG) rely faithfully encode information related to physical properties. To this end, the benchmark introduces a dataset covering a wide range of physical property families, from fluid, solid mechanics, and dynamics to optics. Based on this dataset, the benchmark evaluates how well the representation learning methods capture physical properties through three tasks: text-video retrieval, physical property regression, and pair classification.

Critical strengths

According to the paper, unlike existing similar benchmarks, the proposed benchmark enables the evaluation and analysis of a diverse range of physical properties, from fluid, solid mechanics, and dynamics to optics, while also considering physical quantities. This allows for a more fine-grained evaluation of physical properties beyond simply assessing scene semantics. The tree evaluation tasks seem to further enable the physical understanding of representation learning methods to be evaluated and analyzed from multiple aspects.

Critical weaknesses

CW1. Regarding the classification task utilized in the paper, could the classification performance of a model (for both within- and cross-family tasks) be achieved through an easy answer that does not require model to understand physical properties, but instead relies on simple visual difference? For example, within-family classification may be possible simply by recognizing difference in objects in the solid mechanics family. Moreover, cross-family classification may be even easier, as the classes could potentially be distinguished based on visually apparent differences in image statistics, which could explain the higher classification accuracy compared to within-family classification (in Table 2). If this is the case, does high classification accuracy necessarily indicate that a model has a better understanding of physical properties, or could it instead reflect the model’s ability to recognize superficial visual differences between the classes?

CW2. Regarding the retrieval task, could the near-chance-level performance of most models be attributed not to a lack of understanding of physical properties, but simply to poor generalization to out-of-distribution data? For example, most models are trained primarily on natural videos. Therefore, their poor performance on the simulated data in this benchmark, which differs substantially from model’s training data, could instead reflect limited robustness to out-of-distribution data rather than an inability to understand physical properties.

Questions for authors

As pointed out in the critical weaknesses above, I believe that two of three tasks proposed by the paper for evaluating models’ physical knowledge, namely retrieval and classification, may be confounded by factors unrelated to physical knowledge. Therefore, I would like to know whether the benchmark provides any analysis or interpretation that can account for these factors and help disentangle their effects from the models’ actual understanding of physical properties.

4 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer EKUt

Thank you for the careful evaluation of our work and the insightful questions. We appreciate the opportunity to clarify the scope of our benchmark and the complementary roles of the three evaluation tasks. We address your concerns point-by-point as follows:

1. Superficial Visual Differences in Pair Classification

Thanks for the comment. We agree that visual cues may contribute to cross-family evaluation, and would like to clarify that within-family and cross-family classification are designed to evaluate different levels of physical discrimination.

Cross-family classification evaluates discrimination across different physical systems. We acknowledge that coarse visual and semantic differences between families may contribute to performance in this setting. Therefore, high cross-family accuracy alone should not be interpreted as evidence of fine-grained physical understanding.

Within-family classification, on the other hand, uses negative examples from the same simulation family. The positive and negative cases share the same underlying physical system and broad scene characteristics, but differ in their case-specific conditions and physical responses. This design reduces reliance on simple scene-category recognition and provides a more challenging evaluation of fine-grained video-text correspondence. While visual cues may still contribute, this setting is less susceptible to the coarse visual differences present in cross-family classification.

Importantly, the substantial performance gap between these two settings is itself informative. As shown in Table 2, pretrained embedding models achieve relatively strong cross-family classification but remain close to chance on within-family classification. This suggests that existing embeddings are substantially better at distinguishing broad physical scenarios than at aligning fine-grained physical distinctions.

We also emphasize that pair classification is only one of three complementary evaluation tasks. Our retrieval evaluation requires matching detailed physical descriptions against larger video candidate pools (700-2700 videos depending on the specific physics branch), while physical-property regression directly measures the recoverability of continuous physical quantities from video embeddings. These tasks provide additional perspectives beyond categorical visual discrimination.

2. Retrieval and Out-of-distribution Generalization

We agree that the distribution gap between natural and simulated videos may contribute to the low zero-shot retrieval performance. However, our findings suggest that the issue is not simply an inability to extract meaningful physical information from simulated videos.

First, as shown in our regression evaluation, pretrained video representations already support quantitative physical-property prediction through lightweight linear probes. In particular, frozen LCO-Embedding-7B representations achieve an average nRMSE of 0.0202 across the ten regression families. This indicates that meaningful physical information is recoverable from simulated videos, even without physics-specific representation training.

Second, physics-aware contrastive adaptation substantially improves retrieval performance on the same benchmark evaluation distribution. For example, LCO-Embedding-7B improves Fluid R@10 from 4.9% to 17.9% and Dynamics R@10 from 5.7% to 21.0% after physics adaptation. These results demonstrate that cross-modal alignment with physical descriptions can be substantially improved without changing the evaluation distribution. The improvements would have been more difficult if the knowledge is not encoded in the base foundation models, according to studies on the relationship between base model capabilities and their representation quality after contrastive learning.

These findings suggest that physical information can be encoded in pretrained representations without being effectively exposed through embedding similarity. We acknowledge that domain shift may contribute to the observed retrieval difficulties, and that physics-specific adaptation may also improve familiarity with the simulation domain. Therefore, we do not interpret low zero-shot retrieval performance as definitive evidence that pretrained models lack physical knowledge. Instead, our benchmark aims to characterize how effectively current representations capture and expose fine-grained physical information under a controlled evaluation setting.

AuthorsOpenReview ↗

(2/2) Response to Reviewer EKUt

3. Disentangling confounding factors across tasks

Thanks for this comment. A central motivation of our benchmark is to evaluate physical representations from multiple complementary perspectives, rather than using a single task as a definitive measure of physical understanding and reasoning.

As described in Section 2.2, the three tasks assess different capabilities:

Classification: Cross-family classification primarily measures coarse-grained discrimination between physical systems, where visual differences may contribute to performance. Within-family classification reduces this confounding factor by comparing instances of the same physical system with different conditions and responses, providing a more fine-grained evaluation.

Retrieval: Retrieval evaluates the alignment between detailed physical descriptions and video representations. While it provides a more challenging measure of fine-grained cross-modal correspondence, its performance may also be affected by visual differences and domain shift.

Regression: Regression provides a more direct quantitative evaluation of physical-property recoverability. We use controlled simulations in which the target physical property varies while other relevant conditions remain fixed, allowing us to examine whether the corresponding information can be recovered from frozen video representations.

Importantly, the discrepancies across these tasks are themselves informative. Pretrained models can achieve relatively strong quantitative regression performance while remaining close to chance on within-family classification and performing poorly on retrieval. This suggests that quantitative information recoverability, fine-grained discrimination, and cross-modal alignment are distinct capabilities that should not be conflated. The complementary evaluations help characterize these differences and reveal limitations that would be obscured by a single task.

Thank you for the response. The authors acknowledge that visual differences may contribute to cross-family classification and that domain shift may contribute to the low retrieval performance. However, no additional analysis is provided to separate these factors from physical understanding. In particular, within-family cases can still differ in visible properties such as object geometry, and both the regression probes and physics adaptation rely on simulated data, leaving the effect of domain shift unexamined. For these reasons, I will maintain my score.

Thank you to the authors for the detailed rebuttal and additional experiments, which addressed several of my concerns.

However, my primary concern remains. While the pixel-based regression results provide useful evidence of recoverability under controlled conditions, they do not fully establish that individual video-text pairs are uniquely distinguishable, especially when different physical configurations producing similar observations.

I appreciate the additional clarifications and experiments, but I believe this fundamental concern remains insufficiently resolved. Therefore, I will maintain my current rating.

Reviewer_yA5gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the World Embedding Benchmark for evaluating physical information in video representations using simulated videos paired with physical annotations and descriptions. It combines retrieval, physical property regression, and video description matching to distinguish language alignment from the recoverability of quantitative information.

Critical strengths
  • The paper makes a useful distinction between encoding physical information and aligning that information with language. Evaluating both retrieval and property regression reveals differences that either task alone could overlook.

  • The use of parameterized simulations provides a clear connection between each video and its generating physical conditions. Organizing related cases into families enables comparisons between closely related physical scenarios. This gives the evaluation an interpretable structure and reduces reliance on subjective judgments of physical plausibility.

Critical weaknesses
  • My biggest concern is that the benchmark does not sufficiently establish whether the physical distinctions required by its descriptions are identifiable from the rendered videos. Different combinations of material properties, loading, and scale can produce similar visible behavior. Without clarifying which distinctions remain observable, retrieval failures cannot be attributed entirely to weak physical representations.

  • More specifically, both texts and videos often contain ambiguous info. A description may specify physical parameters that cannot be uniquely inferred from the rendered behavior, leaving multiple descriptions compatible with the same visual evidence. Treating only the original simulation pair as correct may therefore penalize physically reasonable matches and overstate the model’s limitations.

  • Also, the tradeoff between alignment and regression is also underdetermined. Reduced linear prediction accuracy after adaptation could reflect a change in how information is organized, even if that information remains recoverable through a different mapping.

Questions for authors

How are the 400 probe training examples per family in Figure 6 generated and separated from the benchmark test cases and contrastive adaptation data?

How would retrieval change if a query expressed the same physical conditions using different units or a paraphrased description while keeping the target video unchanged?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yA5g (1/4) — point 1: visual identifiability (part 1/2)

Thank you for your thoughtful and insightful comments. We appreciate the opportunity to provide clarifications and additional experiments in our responses below.

1. (point 1 part 1/2) Visual Identifiability

We appreciate the reviewer's concern regarding the visual identifiability of physical properties. We address this question from two complementary perspectives: (1) the controlled simulation design and detailed retrieval queries in our original benchmark, and (2) additional pixel-based experiments conducted to assess visual recoverability.

(1) Controlled benchmark design and query specificity.

A key motivation for using physics-based simulations is to obtain precise physical parameters and ground-truth annotations under controlled conditions. Moreover, our retrieval queries jointly specify the physical scenario, underlying mechanism, visualization, case-specific conditions, and quantitative diagnostics, rather than relying on a single potentially ambiguous property.

As detailed in Appendix C.1 (Figure 5), the actual queries are substantially more specific than the simplified descriptions illustrated in Figure 1 in the main paper. We provide two examples directly from our retrieval task:

Example 1 — Articulated Chain (Dynamics)

Find the 2D dynamics video of 3 link articulated pendulum chain, showing a chain of hinged rigid links swinging under gravity or a root drive. Physically, joint constraints transmit motion along the chain, creating phase lag, folding, or free-swing response. The color map represents link and joint speed magnitude along the chain, so speed colors show which parts move fastest. Use the variant with drive mode set to free release, link count set to short, terminal mass set to low; Earth-scale gravity acts downward; no separate ground plane is part of the visible setup. To identify the exact video, look for the mechanism rotates through about 20.81 degrees, linked bodies reach a relative angle near 22.45 degrees, the fastest visible motion reaches about 2.084 m/s, the recorded motion covers roughly 6.64 s.

Example 2 — Beam Splitter (Optics)

Find the 2D optics video of beam-splitter plate branching, showing one incident beam dividing into reflected and transmitted branches. Physically, the plate divides the incident energy between a reflected and a transmitted branch whose balance depends on the plate geometry. The color map represents the normalized irradiance field, so the two outgoing branches carry complementary brightness levels. Use the variant where incident-angle trim moves from about -4.9 deg to about 7.1 deg across 32 sampled frames. To identify the exact video, look for the splitter balance error is 0.050142, the branch separation is 1.1768, and the mean transmission efficiency is 0.5251.

These examples illustrate that retrieval queries combine multiple physical conditions and observable responses, providing complementary cues for distinguishing simulation instances.

(2) Additional pixel-based evaluation of visual recoverability.

To further investigate visual recoverability, we conducted additional regression experiments using raw RGB and grayscale pixel features, without pretrained visual encoders.

We use the regression task of the paper covering ten physical families, with 400 training and 100 test examples per family. Within each family, the target physical property varies while other relevant conditions are controlled. We train family-specific linear regressors using different temporal sampling configurations, with training-only feature normalization and 4-fold cross-validation for dimension-aware regularization selection.

Table 1. Summary of pixel-based regression results. Average nRMSE (×100, ↓) across ten families.

RepresentationTemporal InputFeature DimensionAverage nRMSE ↓
LCO-3BFull video2,0482.47
RGB pixelsUniformly sampled 4 frames2,764,8001.03
RGB pixelsUniformly sampled 8 frames5,529,6000.78
Grayscale pixelsUniformly sampled 4 frames921,6001.62
Grayscale pixelsUniformly sampled 8 frames1,843,2001.30

These results provide additional empirical support for the visual recoverability of quantitative physical properties in our controlled simulation settings. Even simple linear regression on pixel features achieves average nRMSEs of 0.78 and 1.30 using 8-frame RGB and grayscale inputs, respectively.

We provide the complete per-family results across all temporal sampling configurations in Point 1 (2/2) below.

AuthorsOpenReview ↗

Response to Reviewer yA5g (2/4) — point 1: visual identifiability (part 2/2)

1. (point 1 part 2/2) Visual Identifiability

Continuing the pixel-based visual recoverability analysis in Point 1 (1/2), we report the full per-family results below, including single-frame, two-frame, and uniformly sampled multi-frame inputs.

Table 2. RGB pixel-based physical-property regression (nRMSE ×100, ↓).

FamilyLCO-3BFirstLastFirst+LastUniform 4Uniform 8
Feature Dimension2,048691,200691,2001,382,4002,764,8005,529,600
Dye 2D1.954.871.471.491.521.39
Flow Past Cylinder 2D2.680.480.510.480.350.35
Cylinder Torsion 3D2.641.400.650.710.560.53
Hyperelastic Cuboid 3D2.440.950.390.470.320.27
Long Cuboid Bending 3D3.020.740.460.410.320.29
Pendulum2.132.079.072.673.482.41
Rigid Collision2.0114.413.693.280.400.52
Sphere Drop 3D2.0515.4611.7711.592.781.51
Double-Slit Interference2.020.230.200.200.200.19
Total Internal Reflection3.770.290.540.350.390.36
Average2.474.092.882.171.030.78

Table 3. Grayscale pixel-based physical-property regression (nRMSE ×100, ↓).

FamilyFirstLastFirst+LastUniform 4Uniform 8
Feature Dimension230,400230,400460,800921,6001,843,200
Dye 2D5.811.421.441.391.33
Flow Past Cylinder 2D0.700.920.790.640.60
Cylinder Torsion 3D1.671.190.810.740.78
Hyperelastic Cuboid 3D0.920.360.430.320.29
Long Cuboid Bending 3D0.850.510.490.420.39
Pendulum2.229.773.256.574.33
Rigid Collision14.4211.8915.050.780.96
Sphere Drop 3D15.4216.6714.254.473.45
Double-Slit Interference0.380.270.240.170.15
Total Internal Reflection0.560.880.540.740.68
Average4.304.393.731.621.30

The per-family results further highlight the importance of temporal information for dynamic phenomena. For instance, RGB nRMSE decreases from 14.41 to 0.40 for rigid collision when moving from the first frame to four uniformly sampled frames, and from 15.46 to 1.51 for sphere dropping when using eight frames.

We emphasize that this experiment serves as a diagnostic of visual recoverability, rather than a direct comparison with embedding model-based methods. Raw pixel features retain substantially higher-dimensional information (up to 5.53M dimensions) than the compact 2,048-dimensional LCO embedding representing the full video.

Overall, our controlled simulation design and detailed multi-constraint retrieval queries, together with the additional pixel-based experiments, provide complementary evidence that meaningful physical information is visually recoverable from the rendered videos. These findings support World Embedding Benchmark as a controlled testbed for evaluating fine-grained physical representations.

AuthorsOpenReview ↗

Response to Reviewer yA5g (3/4)

2. (On Alignment-Recoverability Trade-off) Information remains recoverable through a different mapping after contrastive learning?

Thanks for this comment. Our original regression evaluation uses linear probing to measure the recoverability of quantitative physical properties from frozen embeddings. To further assess whether the observed degradation is specific to linear readout, we have extended our experiments to nonlinear probing using RBF Kernel Ridge Regression.

Furthermore, we additionally performed physics-aware contrastive adaptation on Qwen3-VL-Embedding-2B, extending analysis beyond LCO-Embedding to different model families. The following displays results of all three backbones (LCO-3B/7B and Qwen3-VL-2B), both before and after physics adaptation, using linear and nonlinear probes across all ten regression subtasks.

Table 1. Linear and nonlinear regression before and after physics adaptation. Results report nRMSE (×100, ↓), averaged within each physics branch and across all ten subtasks. Note that the lower is better for regression errors.

ModelProbeFluidSolidDynamicsOptics & EM10-task Avg.
LCO-3BLinear Regression2.312.702.062.902.47
LCO-3B-adaptedLinear Regression3.37 (+1.06 ↑)4.88 (+2.18 ↑)3.37 (+1.31 ↑)3.99 (+1.09 ↑)3.95 (+1.48 ↑)
LCO-3BRBF Kernel Regression1.842.161.602.782.05
LCO-3B-adaptedRBF Kernel Regression3.12 (+1.28 ↑)4.51 (+2.35 ↑)2.51 (+0.91 ↑)3.79 (+1.01 ↑)3.49 (+1.44 ↑)
LCO-7BLinear Regression2.012.021.692.542.02
LCO-7B-adaptedLinear Regression2.75 (+0.74 ↑)3.17 (+1.15 ↑)2.62 (+0.93 ↑)3.03 (+0.49 ↑)2.89 (+0.87 ↑)
LCO-7BRBF Kernel Regression1.831.891.382.461.84
LCO-7B-adaptedRBF Kernel Regression2.32 (+0.49 ↑)4.17 (+2.28 ↑)2.25 (+0.87 ↑)2.95 (+0.49 ↑)2.98 (+1.14 ↑)
Qwen3-VL-2BLinear Regression1.731.901.561.911.76
Qwen3-VL-2B-adaptedLinear Regression3.05 (+1.32 ↑)3.42 (+1.52 ↑)3.46 (+1.90 ↑)3.40 (+1.49 ↑)3.36 (+1.60 ↑)
Qwen3-VL-2BRBF Kernel Regression1.561.571.101.781.47
Qwen3-VL-2B-adaptedRBF Kernel Regression2.58 (+1.02 ↑)3.49 (+1.92 ↑)2.60 (+1.50 ↑)3.42 (+1.64 ↑)3.03 (+1.56 ↑)

The results show that nonlinear probing can improve quantitative recoverability, particularly for the original representations. However, the regression degradation following physics adaptation persists under nonlinear probing across all three backbones and all four physics branches (12/12 comparisons). Specifically, the average nRMSE under RBF KRR increases from 2.05 to 3.49 for LCO-3B, from 1.84 to 2.98 for LCO-7B, and from 1.47 to 3.03 for Qwen3-VL-2B.

These findings demonstrate that the observed degradation is not limited to linear readout and remains consistent across different model families and probing methods. While more expressive mappings may further improve recoverability, the empirical trade-off between retrieval alignment and quantitative recoverability persists under both linear and nonlinear probing in our experiments.

AuthorsOpenReview ↗

Response to Reviewer yA5g (4/4)

3. Regression Dataset Construction

Thanks for this question. As described in Section 2.2 and Appendix C.2 (Table 5), our regression benchmark consists of 10 family-specific tasks, each targeting a continuous physical property with a specified range.

To clarify the data construction, the regression dataset is independently generated from the retrieval benchmark, using a controlled-variable design. For each selected family, we generate 500 simulation examples, fixing the geometry, initial and boundary conditions, and other relevant parameters while varying only the target physical property. For example:

  • Pendulum: Gravity varies from 2–18 m/s², affecting the oscillation period and phase.
  • Rigid collision: Object mass varies from 2–18 kg, affecting post-collision velocities.
  • Double-slit interference: Wavelength varies from 450–700 nm, affecting fringe spacing.
  • Cuboid bending: Young's modulus varies from 5–50 GPa, affecting beam deflection.

For each family, 400 examples are used for probe training and 100 are held out for testing. These regression examples are generated separately from the retrieval evaluation and contrastive adaptation data, with no shared video instances. For the probe-data scaling experiments in Figure 6, smaller training subsets are strictly nested within larger subsets, while the test set remains fixed, ensuring consistent comparisons across training sizes. We will make these dataset construction and splitting details more explicit in Appendix C.2 of the revised paper.

4. Robustness to Query Rephrasing and Unit Conversion

We evaluated the robustness of retrieval to equivalent physical descriptions through unit conversion and query reformulation.

(1) Robustness to unit conversion.

For the Rolling-Sliding Transition family, we consistently converted all relevant physical quantities (length: m → cm; velocity: m/s → km/h; time: s → ms), keeping the remaining query content and video candidates unchanged.

Table 1. Unit Conversion

ModelMedian Rank (Original → Converted) ↓Query Embedding Cosine Similarity ↑Same Top-1 Video (%) ↑
LCO-Embedding-3B76 → 770.99898%
LCO-Embedding-7B54 → 540.99794%
Qwen3-VL-Embedding-2B52 → 500.99781%

The results show robustness to unit conversion. Across all three models, query embedding cosine similarity is at least 0.997, and median retrieval rank changes by at most two positions. Also, 81%–98% retrieve the same top-ranked video.

(2) Robustness to controlled query reformulation.

We also evaluated two types of controlled paraphrasing across all 80 families (8,000 queries) and four physics categories:

  • A1: Lexical reformulation: Replacing fixed template expressions with semantically equivalent alternatives, while keeping all physical descriptions, numerical values, and units unchanged.
  • A2: Structural reformulation: Reordering the descriptive sentences and the numerical attributes within the final sentence, while preserving the complete original content.

We evaluated three embedding models using the same video candidates and retrieval protocol. The following results report R@10 (%) averaged over text-to-video and video-to-text retrieval, with ranking performed over the full video candidate pool of each category. Values in parentheses indicate absolute changes relative to the original queries (percentage points).

Table 2. Controlled Paraphrasing

ModelQueryFluidSolidDynamicsOptics & EM
LCO-Embedding-3BOriginal3.91.55.10.7
A1: Lexical3.9 (+0.1)1.5 (+0.0)4.1 (-0.9)0.7 (-0.0)
A2: Structural3.6 (-0.2)1.5 (+0.0)4.9 (-0.2)0.7 (+0.0)
LCO-Embedding-7BOriginal4.91.65.71.9
A1: Lexical4.0 (-0.9)1.6 (+0.0)6.0 (+0.3)2.0 (+0.0)
A2: Structural4.5 (-0.4)1.6 (-0.1)5.3 (-0.4)2.1 (+0.1)
Qwen3-VL-Embedding-2BOriginal3.91.67.32.3
A1: Lexical3.3 (-0.6)1.5 (-0.1)7.4 (+0.1)2.2 (-0.2)
A2: Structural3.9 (-0.1)1.7 (+0.1)7.3 (+0.0)2.2 (-0.1)

Overall, retrieval performance remains largely stable under both transformations. Across the 24 model-category comparisons, 21 exhibit changes of no more than 0.4 percentage points, with the largest change being 0.9 percentage points. Importantly, the changes do not exhibit a consistent degradation pattern across models or physics categories.

Taken together, these experiments provide complementary evidence that our retrieval results are not strongly dependent on a particular unit convention, fixed template vocabulary, or sentence ordering. The unit-conversion experiment demonstrates consistency under equivalent numerical representations, while the experiments across 80 families further establish robustness to controlled lexical and structural variations in physical descriptions.

Reviewer_9BD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a world embedding benchmark to evaluate the physical information preservation ability in current video representations. The proposed benchmark consists of 8k simulation cases from four domains, i.e., fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. It defines three evaluation tasks, including text-video retrieval, physical property regression, and pair classification. Several models have been evaluated on the proposed benchmark, such as Qwen, V-JEPA, and Video-CLIP-XL. This paper also discovers a few interesting phenomena, such as that quantitative physical information can be recovered from video representations via simple linear regression probes, and that physical embeddings can be leveraged for retrieval-augmented video generation.

Critical strengths
  1. This paper provides a novel insight that current video representations carry predictive physical properties (via regression probes), but they may not be accessible via cross‑modal similarity (validated through the retrieval and pair-classification experiments).

  2. The proposed simulation dataset is comprehensive and covers several physics solvers (e.g., OpenFOAM).

  3. This paper also validates the potential of physical embeddings in downstream applications like video generation.

Critical weaknesses
  1. My major concern is the simulation-to-real gap. As all videos are clean simulation results, whether the findings in this paper can be applied to complex, real-world videos is unclear.

  2. Some selected models are not evaluated thoroughly. For instance, MLLMs are evaluated only on the pairwise classification task. Why can't we use their hidden states for regression and retrieval?

  3. The observation in Sec. 3.2 suggests a trade‑off between alignment and quantitative information recoverability. However, this is shown on the LCO-Embedding model, which does not mean it is a general issue of current models or contrastive learning,

Questions for authors
  1. Is the observed trade‑off general across embedding architectures?

  2. In the retrieval-augmented video generation experiment, why are only optics and thermal considered? What metrics are used in this experiment?

  3. Will the full simulation pipeline and benchmark be released?

3 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer 9BD5

Thank you for the careful reading of our paper and the constructive feedback. We appreciate the opportunity to clarify the methodology and provide additional experiments. We address your concerns as follows:

1. Simulation-to-real gap

Our benchmark is primarily designed as a controlled testbed for evaluating physical information in video embeddings. Simulations provide precise physical ground truth, whereas annotating physical properties and dynamics in real-world videos is often difficult and inevitably introduces information loss.

Importantly, Section 3.3 also provides evidence of transfer beyond simulation. Models adapted using simulated videos are evaluated on a retrieval-augmented video generation task involving real-world physical phenomena and reference videos. The physics-adapted model improves the PhyGenBench score from 0.63 to 0.66 over the original model (Table 3), suggesting that the learned representations can also benefit real-world applications.

We agree that broader evaluation on real-world videos would be valuable and will clarify this limitation in the paper.

2. Evaluation of generative MLLMs

Thanks for the comment. Our original evaluation was designed to assess each model family using its most suitable evaluation protocol: dedicated embedding models were evaluated through embedding-based retrieval, regression, and pair classification, while generative MLLMs were evaluated through direct prompting.

Without contrastive learning-based training, representations of pre-trained generative models (such as MLLMs) are known to be highly anisotropic and poorly suited for direct similarity-based retrieval. However, their hidden states may still preserve useful information recoverable through linear probing.

Following the reviewer's suggestion, we have extended the evaluation to all four Qwen3.5/3.6 models using their hidden states, with Qwen3-VL-Embedding-8B and LCO-Embedding-7B included as embedding baselines for comparison.

ModelRetrieval R@10 (F / S / D / O) ↑Regression nRMSE (×100) ↓
Qwen3-VL-Embedding-8B6.6 / 1.9 / 7.4 / 3.11.80
LCO-Embedding-7B4.9 / 1.6 / 5.7 / 1.92.02
Qwen3.5-27B3.6 / 0.7 / 4.6 / 0.82.04
Qwen3.5-35B-A3B2.6 / 0.9 / 4.4 / 1.12.72
Qwen3.6-27B4.1 / 0.5 / 3.6 / 1.12.23
Qwen3.6-35B-A3B2.8 / 0.6 / 5.1 / 1.02.56

The results show that MLLM hidden states retain substantial physical information, achieving regression performance comparable to dedicated embedding models (nRMSE ×100: 2.04–2.72). However, their retrieval performance remains consistently weaker, suggesting that the encoded information is not necessarily well aligned for cross-modal similarity matching.

We further compare hidden-state similarity against direct prompting on the same pair-classification tasks, providing a direct comparison of the two evaluation protocols.

ModelHidden-state Similarity (Within / Cross) ↑Direct Prompting (Within / Cross) ↑
Qwen3-VL-Embedding-8B53.1 / 86.2—
LCO-Embedding-7B46.8 / 76.4—
Qwen3.5-27B50.1 / 69.061.3 / 92.0
Qwen3.5-35B-A3B50.0 / 66.558.8 / 91.2
Qwen3.6-27B48.9 / 67.160.2 / 91.9
Qwen3.6-35B-A3B52.8 / 65.059.0 / 90.9

These results reveal a substantial gap between hidden-state similarity and direct prompting for generative MLLMs. In particular, direct prompting consistently outperforms hidden-state similarity by 23.0–25.9 percentage points on cross-family classification. This further supports our choice of prompt-based evaluation for generative MLLMs, and highlights that physical information accessible through generative reasoning is not necessarily reflected in their embedding similarity.

We will incorporate these additional results and clarify the evaluation protocols in the revised paper.

AuthorsOpenReview ↗

(2/3) Response to Reviewer 9BD5

3. Alignment-recoverability trade-off across model families

We thank the reviewer for this comment. We have extended our experiments across additional model families and training data scales, including physics-aware contrastive adaptation of Qwen3-VL-Embedding-2B using 2K, 4K, and 8K physics video-text pairs.

The extended results show a consistent pattern across two model families, three model backbones, and nine adapted checkpoints. As summarized in Tables 1 and 2 below, physics adaptation consistently improves retrieval performance across all four physics branches, while increasing regression error relative to the corresponding original models.

Table 1. Performance before and after physics adaptation (8K training examples). Changes relative to the original models are shown in parentheses. Note that higher nRMSE indicates worse regression performance.

ModelRetrieval R@10 (F / S / D / O) ↑Regression nRMSE (×100) ↓Pair Classification (Within / Cross) ↑
Qwen3-VL-Embedding-2B3.9 / 1.6 / 7.3 / 2.31.7651.5 / 80.8
+ Physics (8K)18.3 (+14.4 ↑) / 6.2 (+4.6 ↑) / 17.0 (+9.7 ↑) / 8.5 (+6.2 ↑)3.36 (+1.60 ↑)51.6 (+0.1 ↑) / 97.5 (+16.7 ↑)
LCO-Embedding-3B3.9 / 1.5 / 5.1 / 0.72.4751.5 / 68.5
+ Physics (8K)18.6 (+14.7 ↑) / 8.0 (+6.5 ↑) / 16.5 (+11.4 ↑) / 8.9 (+8.2 ↑)3.95 (+1.48 ↑)53.5 (+2.0 ↑) / 98.2 (+29.7 ↑)
LCO-Embedding-7B4.9 / 1.6 / 5.7 / 1.92.0246.8 / 76.4
+ Physics (8K)17.9 (+13.0 ↑) / 9.4 (+7.8 ↑) / 21.0 (+15.3 ↑) / 9.4 (+7.5 ↑)2.89 (+0.87 ↑)59.6 (+12.8 ↑) / 98.4 (+22.0 ↑)

We further provide the complete results across all three training scales for each backbone in Table 2, together with the corresponding original model baselines.

Table 2. Effects of physics adaptation across model families and training data scales. Parentheses indicate absolute changes relative to the corresponding original model. Retrieval reports R@10 (Fluid / Solid / Dynamics / Optics), regression reports average nRMSE (×100), and pair classification reports accuracy (%).

ModelTraining DataRetrieval R@10 (F / S / D / O) ↑Regression nRMSE ↓Pair Classification (Within / Cross) ↑
Qwen3-VL-Embedding-2BOriginal3.9 / 1.6 / 7.3 / 2.31.7651.5 / 80.8
Qwen3-VL-Embedding-2BPhysics 2K18.1 (+14.2 ↑) / 7.3 (+5.7 ↑) / 19.2 (+11.9 ↑) / 6.7 (+4.4 ↑)3.23 (+1.47 ↑)55.4 (+3.9 ↑) / 97.0 (+16.2 ↑)
Qwen3-VL-Embedding-2BPhysics 4K20.5 (+16.6 ↑) / 7.7 (+6.1 ↑) / 19.3 (+12.0 ↑) / 8.5 (+6.2 ↑)3.18 (+1.42 ↑)54.4 (+2.9 ↑) / 97.8 (+17.0 ↑)
Qwen3-VL-Embedding-2BPhysics 8K18.3 (+14.4 ↑) / 6.2 (+4.6 ↑) / 17.0 (+9.7 ↑) / 8.5 (+6.2 ↑)3.36 (+1.60 ↑)51.6 (+0.1 ↑) / 97.5 (+16.7 ↑)
LCO-Embedding-3BOriginal3.9 / 1.5 / 5.1 / 0.72.4751.5 / 68.5
LCO-Embedding-3BPhysics 2K16.4 (+12.5 ↑) / 8.1 (+6.6 ↑) / 17.2 (+12.1 ↑) / 7.3 (+6.6 ↑)2.86 (+0.39 ↑)55.0 (+3.5 ↑) / 98.2 (+29.7 ↑)
LCO-Embedding-3BPhysics 4K17.6 (+13.7 ↑) / 7.1 (+5.6 ↑) / 18.0 (+12.9 ↑) / 9.0 (+8.3 ↑)3.23 (+0.76 ↑)53.1 (+1.6 ↑) / 98.9 (+30.4 ↑)
LCO-Embedding-3BPhysics 8K18.6 (+14.7 ↑) / 8.0 (+6.5 ↑) / 16.5 (+11.4 ↑) / 8.9 (+8.2 ↑)3.95 (+1.48 ↑)53.5 (+2.0 ↑) / 98.2 (+29.7 ↑)
LCO-Embedding-7BOriginal4.9 / 1.6 / 5.7 / 1.92.0246.8 / 76.4
LCO-Embedding-7BPhysics 2K21.8 (+16.9 ↑) / 11.4 (+9.8 ↑) / 19.9 (+14.2 ↑) / 8.1 (+6.2 ↑)2.50 (+0.48 ↑)58.6 (+11.8 ↑) / 98.5 (+22.1 ↑)
LCO-Embedding-7BPhysics 4K16.3 (+11.4 ↑) / 7.4 (+5.8 ↑) / 17.8 (+12.1 ↑) / 8.8 (+6.9 ↑)2.26 (+0.24 ↑)53.6 (+6.8 ↑) / 98.0 (+21.6 ↑)
LCO-Embedding-7BPhysics 8K17.9 (+13.0 ↑) / 9.4 (+7.8 ↑) / 21.0 (+15.3 ↑) / 9.4 (+7.5 ↑)2.89 (+0.87 ↑)59.6 (+12.8 ↑) / 98.4 (+22.0 ↑)

Across all nine adapted checkpoints, retrieval performance improves in every physics branch, while regression error consistently increases relative to the corresponding original model. This pattern holds across different model families and adaptation scales, although the magnitude of the changes is not monotonic with training data size.

These results further substantiate our observation of the trade-off that improved cross-modal alignment does not necessarily imply better quantitative information recoverability. While we do not claim that this trade-off is universal to contrastive learning, the consistent results across multiple model families and training configurations demonstrate that the phenomenon is not specific to LCO-Embedding. We will incorporate these additional experiments into the revised paper.

AuthorsOpenReview ↗

(3/3) Response to Reviewer 9BD5

4. Additional Questions

(1) Scope and Metrics of Retrieval-Augmented Video Generation

Thanks for the question. The retrieval-augmented video generation is intended as a focused downstream validation of physics-aware retrieval, rather than an exhaustive evaluation across all physical domains.

We focus on two complementary PhyGenBench subsets, Optics and Thermal, which involve distinct types of physical phenomena. Optics covers spatially structured processes such as light propagation and optical interactions, while Thermal involves temporally evolving processes such as temperature-driven state transitions. Together, they provide two different settings of examining whether retrieved physical references can help guide video generation. We also note that the effectiveness of retrieval-augmented generation is not necessarily uniform across physical categories. It depends on the relevance of the retrieved reference to the underlying physical process, the baseline capabilities of the generator, and the extent to which the generator can utilize the reference. For example, our results show different improvements across the two subsets, with a larger gain on Thermal than on Optics.

Regarding the evaluation metric, we use PhyGenBench's open-source route evaluation, comprising VQAScore for key physical phenomena detection, LLaVA-Interleave for physical order verification, and InternVideo2 for overall naturalness assessment. The average score improves from 0.60 to 0.66, with improvements of 0.01 in Optics and 0.10 in Thermal.

(2) Release of the benchmark and simulation pipeline

Yes. We will publicly release the full World Embedding Benchmark and its simulation pipeline, including the simulation scripts and configurations, generated videos and physical annotations, retrieval queries, and evaluation code for all three benchmark tasks. This will support full reproducibility and allow the community to generate additional simulation cases, extend existing physical families, and develop new evaluation tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.