- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PhysVerse, a framework for deformable simulation-ready 4D world modeling from monocular videos. The method combines feed-forward scene reconstruction with dense material-property prediction, future video generation, and MPM-based deformable simulation. It predicts depth, camera parameters, dynamic Gaussian primitives, and dense maps of Young’s modulus, Poisson’s ratio, and density, and uses these representations to support both video prediction and physics-based rollout. The method is evaluated on surgical and general robotic datasets. The experiments cover material-property prediction, video generation quality and temporal consistency, deformable simulation, ablation studies on depth and material conditioning, and zero-shot transfer.
Critical strengths
- The framework provides a practical connection between visual reconstruction and deformable simulation. It augments reconstructed geometry with spatially aligned material maps and assigns them to MPM particles in a feed-forward pipeline. Using the same reconstructed depth and material maps for video conditioning provides a useful integration of visual generation and simulation.
- The video-generation experiments demonstrate improvements on both surgical and general-domain robotic datasets. The controlled ablations further show that joint depth and material conditioning improves video fidelity and temporal consistency, supporting the usefulness of the proposed conditioning strategy.
Critical weaknesses
- Section 3.3 constructs material targets by assigning a fixed database triplet to each semantic category. Although the paper acknowledges this limitation, it does not compare against semantic segmentation followed by the same material lookup. The dense-versus-global comparison in Table 2 demonstrates the value of spatially varying assignments, but a segmentation-and-lookup baseline would also produce a dense map. Consequently, the benefit of the proposed Physics Head over this simpler alternative remains unclear.
- The reference rollout uses annotation-derived material values with the same reconstructed geometry, MPM solver, and interaction settings as the evaluated methods. This controlled comparison usefully measures agreement with the reference material assignment, but does not independently validate recovered material properties or deformation against real observations. Moreover, the 3D particle nRMSE measures only terminal positions, leaving intermediate 3D trajectory accuracy unquantified. The paper should distinguish simulator compatibility and reference-rollout agreement from real-world physical accuracy.
Questions for authors
- Could the authors compare the Physics Head against predicted semantic segmentation followed by the same category-level material lookup, using both material-map and simulation metrics? An oracle-segmentation lookup could serve as an upper bound, since it reproduces the target construction.
- The simulation experiment demonstrates that the reconstructed state can be consumed by MPM and produces a deformable rollout. However, since the reference rollout is itself generated with the same solver using annotation-derived material maps, could the authors clarify what aspect of simulation accuracy this experiment is intended to establish? In addition, since the claim concerns deformable rollout, it would be useful to report 3D trajectory errors over the full simulation rather than only at the terminal frame.
- The material head predicts maps that are subsequently lifted to reconstructed particles. Is there any quantitative evaluation of the resulting 3D material field, dynamic Gaussian geometry, or novel-view reconstruction? If not, could the authors clarify why the current evaluation is sufficient to support the claim of a simulation-ready 4D world state?
- The paper states that a VLM is used to generate descriptions of the corresponding interactions. Is the text condition generated from the complete target video, only from the initial observation, or from an independently available action/instruction? If the full future clip is visible when constructing the text condition, the generator would receive privileged information about the future, which would substantially change the interpretation of the future-prediction/world-modeling results.
5 comments
Thank you for the additional experiments. However, my fundamental concerns remain.
Both Seg2Mat and the Physics Head rely on the same category-to-material lookup table, with the latter trained on the corresponding 2D material maps (Eq. 2). Since both methods rely on the same lookup-table-derived 2D material supervision, the authors should analyze intermediate segmentation, classification, and material prediction results to identify the source of the error gap, rather than reporting only final errors.
Second, on PorcineEndo, PhysVerse does not achieve the best results for Young's modulus or density. This reinforces my concern that the method primarily learns category-level material priors tailored to the simulator, rather than reliably estimating real-world physical properties.
Finally, the source and accuracy of the 3D geometry used for trajectory evaluation remain unclear. Please provide quantitative geometric validation, and evaluate the RGB rendered after MPM simulation against real observations. More fundamentally, the paper needs to clarify what constitutes its claimed 4D world representation, as the current experiments do not sufficiently establish a physically consistent 4D reconstruction.
I believe the paper still lacks sufficient analysis, so I will maintain my current score.
Response to Reviewer 7jis [1/2]
We thank the reviewer for the follow-up comments. We provide the requested intermediate analysis, clarify the PorcineEndo comparisons, and narrow the 4D claim below.
1. Material prediction error analysis
Seg2Mat and the Physics Head use the same category-to-material lookup table to construct the 2D supervision, but they differ at inference. Seg2Mat predicts semantic labels and assigns one fixed material triplet to each predicted class. The Physics Head regresses continuous dense material properties in one forward pass, without a segmentation or lookup step.
We first decompose Seg2Mat’s material-prediction error into two sources: mismatch (covered pixels assigned incorrect material values) and uncovered (pixels with no predicted segmentation mask). The percentages show how much each region contributes to Seg2Mat’s total error for each metric. On most dataset–metric pairs, mismatched assignments account for a larger share of Seg2Mat’s error than uncovered regions. Seg2Mat also performs better on some metrics for CaDISv2 and VLA-OS in Table 1, possibly because fixed lookup closely matches the labeling scheme for their limited category inventories.
Material-Prediction Error Breakdown: Seg2Mat
| Dataset | Example Class IoU (%) | Contribution to ALDE_E (%): Mismatch / Uncovered | Contribution to ADE_ν (%): Mismatch / Uncovered | Contribution to ARE_ρ (%): Mismatch / Uncovered |
|---|---|---|---|---|
| CholecSeg8K | Abdominal wall 93.51 | 78.4 / 21.6 | 69.9 / 30.1 | 93.6 / 6.4 |
| EndoVis2018 | Instrument shaft 96.82 | 88.8 / 11.2 | 88.6 / 11.4 | 97.7 / 2.3 |
| CaDISv2 | Pupil 93.51 | 77.8 / 22.2 | 83.0 / 17.0 | 95.8 / 4.2 |
| VLA-OS | Wood 98.29 | 78.2 / 21.8 | 78.2 / 21.8 | 84.6 / 15.4 |
Second, we also analyze the Physics Head’s prediction errors. We divide valid pixels into boundary pixels (within five pixels of a ground-truth material transition) and interior pixels (all remaining valid pixels). The percentages below show each region’s contribution to the Physics Head’s total error for the corresponding metric. Across all four datasets, the Physics Head’s prediction errors are largely concentrated near material boundaries, despite these regions accounting for a relatively small proportion of pixels.
Material-Prediction Error Breakdown: Physics Head
| Dataset | Boundary pixels (%) | Contribution to ALDE_E (%): Boundary / Interior | Contribution to ADE_ν (%): Boundary / Interior | Contribution to ARE_ρ (%): Boundary / Interior |
|---|---|---|---|---|
| CholecSeg8K | 17.32 | 42.2 / 57.8 | 47.0 / 53.0 | 53.6 / 46.4 |
| EndoVis2018 | 14.12 | 61.8 / 38.2 | 67.9 / 32.1 | 78.4 / 21.6 |
| CaDISv2 | 19.80 | 46.0 / 54.0 | 43.7 / 56.3 | 63.2 / 36.8 |
| VLA-OS | 7.35 | 43.4 / 56.6 | 36.4 / 63.6 | 32.3 / 67.7 |
Finally, we compare both methods on the same pixels where Seg2Mat assigns an incorrect material (mismatch) or leaves a pixel without a predicted mask (uncovered). Errors are averaged within each region and then equally across four datasets. Lower is better. Compared with Seg2Mat, the Physics Head achieves lower errors for all three material properties in both regions, with particularly substantial improvements in mismatched regions and more modest gains for in uncovered regions. These results indicate that direct dense prediction mitigates segmentation-related errors by avoiding intermediate segmentation and lookup steps.
Material-prediction errors on regions defined by Seg2Mat.
| Region | Pixels (%) | Method | ALDE_E ↓ | ADE_ν ↓ | ARE_ρ ↓ |
|---|---|---|---|---|---|
| Mismatch | 10.74 | Seg2Mat | 1.2760 | 0.07552 | 0.9027 |
| Physics Head | 0.7325 | 0.04430 | 0.3992 | ||
| Uncovered | 4.78 | Seg2Mat | 2.2171 | 0.05737 | 0.3570 |
| Physics Head | 0.5997 | 0.02628 | 0.3449 |
We thank the reviewer for the helpful suggestions. We have revised the manuscript, and our responses to each point are provided below.
R W1/Q1: Comparison with Segmentation and Material Lookup
We added Seg2Mat, which predicts semantic labels and applies the same fixed category-to-material lookup table used to construct our supervision targets. Oracle segmentation with the same lookup reproduces the category-derived ground truth by construction. As shown in Table 1, our Physics Head performs better on four datasets and remains competitive on CaDISv2 and VLA-OS. We also compare Seg2Mat and our Physics Head in simulation under identical geometry and interaction settings (Table 2). Overall, our dense material maps achieve the lowest full-trajectory error and EPE, while Seg2Mat has a marginally lower terminal nRMSE (Table 2). Unlike Seg2Mat, our Physics Head directly predicts dense material properties in one forward pass without segmentation or lookup at inference time.
Table 1. Material property prediction: Lower is better.
Table 2. Simulation comparison. All variants use the same geometry and simulation settings, differing only in the assigned material maps.
R W2/Q2: Simulation Validation and Trajectory Accuracy
This experiment measures how closely predicted material maps reproduce the annotation-derived reference rollout under identical simulation settings. We additionally report 3D particle trajectory error over the full rollout (Table 2).
To independently validate material prediction accuracy, we further evaluate our method on the PorcineEndo dataset (Table 6), where ground-truth material properties are obtained from real-world physical measurements, providing evidence beyond agreement with annotation-derived references.
Table 6. Material prediction on the PorcineEndo dataset.
R Q3: Evaluation and Simulation-Ready
Our primary focus is material-property estimation, which is evaluated using 2D material maps and simulation metrics. We have added novel-view results, as reported in the table below. By “simulation-ready,” we mean that the evaluated predicted material maps can be mapped to reconstructed geometry produced by the pretrained Gaussian reconstruction head, and these components provide the state needed to run an MPM deformable simulation.
Table 10. Novel-view results on CholecSeg8K and VLA-OS. We evaluate RGB RMSE by reprojecting RGB maps using the rendered depth. Results are averaged over right, left, up, down, forward, and backward views. Lower is better.
R Q4: Text-Conditioned Video Generation
We use a VLM to caption complete clips during dataset construction, as also done in text-conditioned world model training pipelines such as Cosmos and PAN. At evaluation, the caption and initial frame are provided to the generator, while the target frames are used for comparison.