acceptodds
Under review as a conference paper at ICLR 2027

PhysVerse: Deformable Simulation-Ready 4D World Modeling from Monocular Videos.

Abstract

World modeling aims to predict future observations while retaining a scene state for downstream use. Existing reconstruction and generation-centric methods mainly capture geometry and visual dynamics, and rarely represent the spatially varying material parameters a soft-body simulator requires, so the retained state cannot be used directly for deformable simulation. We present PhysVerse, a deformable simulation-ready 4D world model that links reconstruction, video generation, and deformable simulation through a shared scene state. Given monocular video, a feed-forward reconstructor jointly predicts depth, dynamic Gaussian primitives, and dense maps of Young's modulus, Poisson's ratio, and density in a single pass. The reconstructed geometry and material maps initialize MPM-based deformable simulation, while the same depth and material maps condition future video prediction. Generated frames can be passed once again through the reconstructor to recover geometry and material maps. We evaluate PhysVerse on both surgical and general robotic video benchmarks. Experiments show that the reconstructed state supports deformable rollout under MPM, and that depth and material conditioning improve video fidelity and temporal consistency. Our demo is available at: https://anonymous.4open.science/w/doaneqzp-0DBD/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Material Property Prediction, World Model, Deformable Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7jisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhysVerse, a framework for deformable simulation-ready 4D world modeling from monocular videos. The method combines feed-forward scene reconstruction with dense material-property prediction, future video generation, and MPM-based deformable simulation. It predicts depth, camera parameters, dynamic Gaussian primitives, and dense maps of Young’s modulus, Poisson’s ratio, and density, and uses these representations to support both video prediction and physics-based rollout. The method is evaluated on surgical and general robotic datasets. The experiments cover material-property prediction, video generation quality and temporal consistency, deformable simulation, ablation studies on depth and material conditioning, and zero-shot transfer.

Critical strengths
  1. The framework provides a practical connection between visual reconstruction and deformable simulation. It augments reconstructed geometry with spatially aligned material maps and assigns them to MPM particles in a feed-forward pipeline. Using the same reconstructed depth and material maps for video conditioning provides a useful integration of visual generation and simulation.
  2. The video-generation experiments demonstrate improvements on both surgical and general-domain robotic datasets. The controlled ablations further show that joint depth and material conditioning improves video fidelity and temporal consistency, supporting the usefulness of the proposed conditioning strategy.
Critical weaknesses
  1. Section 3.3 constructs material targets by assigning a fixed database triplet (E,ν,ρ)(E,\nu,\rho) to each semantic category. Although the paper acknowledges this limitation, it does not compare against semantic segmentation followed by the same material lookup. The dense-versus-global comparison in Table 2 demonstrates the value of spatially varying assignments, but a segmentation-and-lookup baseline would also produce a dense map. Consequently, the benefit of the proposed Physics Head over this simpler alternative remains unclear.
  2. The reference rollout uses annotation-derived material values with the same reconstructed geometry, MPM solver, and interaction settings as the evaluated methods. This controlled comparison usefully measures agreement with the reference material assignment, but does not independently validate recovered material properties or deformation against real observations. Moreover, the 3D particle nRMSE measures only terminal positions, leaving intermediate 3D trajectory accuracy unquantified. The paper should distinguish simulator compatibility and reference-rollout agreement from real-world physical accuracy.
Questions for authors
  1. Could the authors compare the Physics Head against predicted semantic segmentation followed by the same category-level material lookup, using both material-map and simulation metrics? An oracle-segmentation lookup could serve as an upper bound, since it reproduces the target construction.
  2. The simulation experiment demonstrates that the reconstructed state can be consumed by MPM and produces a deformable rollout. However, since the reference rollout is itself generated with the same solver using annotation-derived material maps, could the authors clarify what aspect of simulation accuracy this experiment is intended to establish? In addition, since the claim concerns deformable rollout, it would be useful to report 3D trajectory errors over the full simulation rather than only at the terminal frame.
  3. The material head predicts H×W×3H\times W\times3 maps that are subsequently lifted to reconstructed particles. Is there any quantitative evaluation of the resulting 3D material field, dynamic Gaussian geometry, or novel-view reconstruction? If not, could the authors clarify why the current evaluation is sufficient to support the claim of a simulation-ready 4D world state?
  4. The paper states that a VLM is used to generate descriptions of the corresponding interactions. Is the text condition generated from the complete target video, only from the initial observation, or from an independently available action/instruction? If the full future clip is visible when constructing the text condition, the generator would receive privileged information about the future, which would substantially change the interpretation of the future-prediction/world-modeling results.
5 comments
AuthorsOpenReview ↗

We thank the reviewer for the helpful suggestions. We have revised the manuscript, and our responses to each point are provided below.

R W1/Q1: Comparison with Segmentation and Material Lookup

We added Seg2Mat, which predicts semantic labels and applies the same fixed category-to-material lookup table used to construct our supervision targets. Oracle segmentation with the same lookup reproduces the category-derived ground truth by construction. As shown in Table 1, our Physics Head performs better on four datasets and remains competitive on CaDISv2 and VLA-OS. We also compare Seg2Mat and our Physics Head in simulation under identical geometry and interaction settings (Table 2). Overall, our dense material maps achieve the lowest full-trajectory error and EPE, while Seg2Mat has a marginally lower terminal nRMSE (Table 2). Unlike Seg2Mat, our Physics Head directly predicts dense material properties in one forward pass without segmentation or lookup at inference time.

Table 1. Material property prediction: Lower is better.

DatasetMethodALDE(E) ↓ADE(ν) ↓ARE(ρ) ↓DatasetMethodALDE(E) ↓ADE(ν) ↓ARE(ρ) ↓
CholecSeg8KPhysVerse (Ours)0.08360.00360.0534VLA-OSPhysVerse (Ours)0.08450.00430.0425
Seg2Mat0.17940.01060.1867Seg2Mat0.03190.00120.0070
EndoVis2018PhysVerse (Ours)0.07200.00350.0741VKittiPhysVerse (Ours)0.24870.01490.1557
Seg2Mat0.07900.00430.1269Seg2Mat0.34320.02190.1698
CaDISv2PhysVerse (Ours)0.09870.00660.0441UrbanSynPhysVerse (Ours)0.14460.00840.0761
Seg2Mat0.04700.00300.0488Seg2Mat0.34240.00930.0838

Table 2. Simulation comparison. All variants use the same geometry and simulation settings, differing only in the assigned material maps.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓End-nRMSE ↓Traj-nRMSE ↓EPE ↓
Grok4.6 (Global)1.36760.03160.115321.44330.43632.6
ChatGPT-5.6 (Global)1.21080.03240.109013.68624.86331.2
Ours (Global)0.28070.01500.08521.1992.50417.5
Seg2Mat (Dense)0.22100.01410.24780.2580.92114.4
Ours (Dense)0.10560.00620.13250.2600.67312.7

R W2/Q2: Simulation Validation and Trajectory Accuracy

This experiment measures how closely predicted material maps reproduce the annotation-derived reference rollout under identical simulation settings. We additionally report 3D particle trajectory error over the full rollout (Table 2).

To independently validate material prediction accuracy, we further evaluate our method on the PorcineEndo dataset (Table 6), where ground-truth material properties are obtained from real-world physical measurements, providing evidence beyond agreement with annotation-derived references.

Table 6. Material prediction on the PorcineEndo dataset.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse (Ours)0.42790.007760.3232
DA3 + Material Head0.40720.009270.2727
VGGT + Material Head0.42940.010420.3602
PhysGM (Lv et al., 2026)0.60360.06810—
VoMP (Dagli et al., 2025)3.65510.116420.3929

R Q3: Evaluation and Simulation-Ready

Our primary focus is material-property estimation, which is evaluated using 2D material maps and simulation metrics. We have added novel-view results, as reported in the table below. By “simulation-ready,” we mean that the evaluated predicted material maps can be mapped to reconstructed geometry produced by the pretrained Gaussian reconstruction head, and these components provide the state needed to run an MPM deformable simulation.

Table 10. Novel-view results on CholecSeg8K and VLA-OS. We evaluate RGB RMSE by reprojecting RGB maps using the rendered depth. Results are averaged over right, left, up, down, forward, and backward views. Lower is better.

DatasetClipPhysVerseVGGTDA3NeoVerse
CholecSeg8Kvideo01_002400.024170.071980.066910.02593
CholecSeg8Kvideo09_008320.012120.058400.054970.01378
CholecSeg8Kvideo12_157500.019120.079120.066010.02689
VLA-OSfold_560.013420.027910.039030.01565
VLA-OSrope_220.012410.019700.034390.01382
VLA-OSjean_770.014830.026570.028740.01546

R Q4: Text-Conditioned Video Generation

We use a VLM to caption complete clips during dataset construction, as also done in text-conditioned world model training pipelines such as Cosmos and PAN. At evaluation, the caption and initial frame are provided to the generator, while the target frames are used for comparison.

Thank you for the additional experiments. However, my fundamental concerns remain.

Both Seg2Mat and the Physics Head rely on the same category-to-material lookup table, with the latter trained on the corresponding 2D material maps (Eq. 2). Since both methods rely on the same lookup-table-derived 2D material supervision, the authors should analyze intermediate segmentation, classification, and material prediction results to identify the source of the error gap, rather than reporting only final errors.

Second, on PorcineEndo, PhysVerse does not achieve the best results for Young's modulus or density. This reinforces my concern that the method primarily learns category-level material priors tailored to the simulator, rather than reliably estimating real-world physical properties.

Finally, the source and accuracy of the 3D geometry used for trajectory evaluation remain unclear. Please provide quantitative geometric validation, and evaluate the RGB rendered after MPM simulation against real observations. More fundamentally, the paper needs to clarify what constitutes its claimed 4D world representation, as the current experiments do not sufficiently establish a physically consistent 4D reconstruction.

I believe the paper still lacks sufficient analysis, so I will maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer 7jis [1/2]

We thank the reviewer for the follow-up comments. We provide the requested intermediate analysis, clarify the PorcineEndo comparisons, and narrow the 4D claim below.

1. Material prediction error analysis

Seg2Mat and the Physics Head use the same category-to-material lookup table to construct the 2D supervision, but they differ at inference. Seg2Mat predicts semantic labels and assigns one fixed material triplet to each predicted class. The Physics Head regresses continuous dense material properties in one forward pass, without a segmentation or lookup step.

We first decompose Seg2Mat’s material-prediction error into two sources: mismatch (covered pixels assigned incorrect material values) and uncovered (pixels with no predicted segmentation mask). The percentages show how much each region contributes to Seg2Mat’s total error for each metric. On most dataset–metric pairs, mismatched assignments account for a larger share of Seg2Mat’s error than uncovered regions. Seg2Mat also performs better on some metrics for CaDISv2 and VLA-OS in Table 1, possibly because fixed lookup closely matches the labeling scheme for their limited category inventories.

Material-Prediction Error Breakdown: Seg2Mat

DatasetExample Class IoU (%)Contribution to ALDE_E (%): Mismatch / UncoveredContribution to ADE_ν (%): Mismatch / UncoveredContribution to ARE_ρ (%): Mismatch / Uncovered
CholecSeg8KAbdominal wall 93.5178.4 / 21.669.9 / 30.193.6 / 6.4
EndoVis2018Instrument shaft 96.8288.8 / 11.288.6 / 11.497.7 / 2.3
CaDISv2Pupil 93.5177.8 / 22.283.0 / 17.095.8 / 4.2
VLA-OSWood 98.2978.2 / 21.878.2 / 21.884.6 / 15.4

Second, we also analyze the Physics Head’s prediction errors. We divide valid pixels into boundary pixels (within five pixels of a ground-truth material transition) and interior pixels (all remaining valid pixels). The percentages below show each region’s contribution to the Physics Head’s total error for the corresponding metric. Across all four datasets, the Physics Head’s prediction errors are largely concentrated near material boundaries, despite these regions accounting for a relatively small proportion of pixels.

Material-Prediction Error Breakdown: Physics Head

DatasetBoundary pixels (%)Contribution to ALDE_E (%): Boundary / InteriorContribution to ADE_ν (%): Boundary / InteriorContribution to ARE_ρ (%): Boundary / Interior
CholecSeg8K17.3242.2 / 57.847.0 / 53.053.6 / 46.4
EndoVis201814.1261.8 / 38.267.9 / 32.178.4 / 21.6
CaDISv219.8046.0 / 54.043.7 / 56.363.2 / 36.8
VLA-OS7.3543.4 / 56.636.4 / 63.632.3 / 67.7

Finally, we compare both methods on the same pixels where Seg2Mat assigns an incorrect material (mismatch) or leaves a pixel without a predicted mask (uncovered). Errors are averaged within each region and then equally across four datasets. Lower is better. Compared with Seg2Mat, the Physics Head achieves lower errors for all three material properties in both regions, with particularly substantial improvements in mismatched regions and more modest gains for ρ\rho in uncovered regions. These results indicate that direct dense prediction mitigates segmentation-related errors by avoiding intermediate segmentation and lookup steps.

Material-prediction errors on regions defined by Seg2Mat.

RegionPixels (%)MethodALDE_E ↓ADE_ν ↓ARE_ρ ↓
Mismatch10.74Seg2Mat1.27600.075520.9027
Physics Head0.73250.044300.3992
Uncovered4.78Seg2Mat2.21710.057370.3570
Physics Head0.59970.026280.3449
Reviewer_7HxfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds a scene representation from monocular video that carries both geometry and material properties, so the same state can drive deformable simulation and future video generation. A material head is trained on top of a frozen HunyuanWorld-Mirror backbone to predict Young's modulus, Poisson's ratio and density, using category-level pseudo-labels built from semantic segmentation and a material database. The predicted materials are assigned to MPM particles for simulation, and, together with first-frame depth, used as control signals for a frozen Wan2.2 model. Material prediction is evaluated on several datasets, video generation on Cholec80-5k and VLA-OS, and simulation by comparing against rollouts driven by the labeled material maps, where dense materials do better than global parameters.

Critical strengths

I think extending scene representations to material fields usable for deformable simulation fits real needs in surgery and deformable-object manipulation, and using the same prediction for both simulation and generation is a direction worth exploring. Besides, the paper reports material prediction, simulation and video generation results. On Cholec80-5k, adding depth and material conditioning improves FVD from 61.04 unconditioned control branch to 55.09, which suggests these signals do help in the current training setup.

Critical weaknesses

1 The "physical material" and "simulation-ready" claims are not independently validated. Material labels are filled in by semantic category, and the simulation reference is generated from the same labels. Matching it shows consistency with the labeling scheme, not with real deformation. There is no "segmentation plus database lookup" baseline, no comparison with observed deformation, and no generation control conditioned on semantic maps. I therefore can't tell whether predicting materials helps beyond the semantic layout. 2 The 4D part and the generation gains are not well supported. It is unclear how a frozen backbone produces dynamic Gaussians with a lifespan parameter. The unconditioned control branch already reaches FVD 61.04, and the paper does not say whether external baselines were adapted the same way. I assume they were not, so the gain over external models cannot all be credited to the material design. 3 The supplementary video does not show what the paper claims. For a 4D paper that claims simulation-ready world modeling, I think the video is where results should be most convincing, since dynamics and 3D consistency are hard to judge from figures. The 26-second video consists of three fixed panels: material maps, a zero-shot EndoNeRF example, and a generation comparison without ground truth. A small 4DGS window is nearly static with no clear novel-view trajectory. There is no MPM simulation, no deformation and no generation-to-reconstruction example. The VLA-OS rope barely moves from 13 s to 23 s, which makes me suspect the VBench consistency scores partly reflect low motion, and dynamic degrees is defined but not reported. I see this as a major weakness, not a presentation detail.

Questions for authors

please refer to major weakness. With the backbone and heads frozen, where does the lifespan parameter come from, and how is it trained? Could you show novel-view renderings and MPM simulations of the actual dynamic Gaussians? Could you compare the material head with segmentation plus database lookup, and try conditioning generation on semantic maps instead? Which checkpoints, data and adaptation budgets were used for the external generation baselines, and could you add the unconditioned-control ablation on VLA-OS? Besides, I feel the related work misses quite a few relevant papers, e.g., Pixie, PhysCtrl, PhysChoreo, PhysWorld and Realistic Surgical Simulation from Monocular Videos. It would be good to cite and briefly discuss them.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7Hxf [1/2]

R W1/Q3:Material Prediction and Simulation

We have added a baseline(Seg2Mat) that predicts semantic segmentation and assigns material properties to each class, as shown in Table 1 in the manuscript. While Seg2Mat performs better on certain datasets, our Physics Head achieves lower errors on others without explicit semantic lookup. We additionally evaluate our material predictions against real-world measurements of physical properties (Table 6), providing an independent comparison beyond our category-derived training labels.

Table 6-1. Material prediction on the PorcineEndo dataset.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse (HWM)0.42790.007760.3232
PhysVerse (DA3)0.40720.009270.2727
PhysVerse (VGGT)0.42940.010420.3602

Table 6-2. Material prediction on the PorcineEndo dataset.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse (Ours)0.42790.007760.3232
PhysGM (Lv et al., 2026)0.60360.06810—
VoMP (Dagli et al., 2025)3.65510.116420.3929

R W2/Q1: Clarification of Gaussian and Ablation

The time-varying Gaussian states are produced at different timestamps by the pretrained WorldMirror reconstruction head, which remains frozen in our method. Across all ablations, the backbone is frozen and the control branch is trained under the same setup.

We additionally evaluate Depth+Material conditioning with Cosmos-H-Surgical. Its official checkpoint was trained on surgical-domain data. The adapted Depth+Material model improves across all reported metrics.

BackboneConditionFVD ↓FID ↓SSIM ↑PSNR ↑LPIPS ↓Subj ↑Mot ↑Flicker ↑Bg ↑
Cosmos-H-SurgicalNone63.1369.930.54416.480.37293.3298.7898.1995.07
Cosmos-H-SurgicalDepth+Material57.4269.150.58817.270.34893.6899.2098.6096.68

R W3: Supplementary Issue

We have added more results on the demo website, including reconstruction results, MPM deformation driven by predicted material properties, and video generation examples. We have also included the dynamic degree metric in Table 4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 7Hxf [2/2]

R Q2: Novel-view and Simulation Results

We provide novel-view evaluation results in Table 10 and MPM deformation visualizations on our project website.

Table 10. Novel-view RGB RMSE. Lower is better.

DatasetClipPhysVerseVGGTDA3NeoVerse
CholecSeg8Kvideo01_002400.024170.071980.066910.02593
CholecSeg8Kvideo09_008320.012120.058400.054970.01378
CholecSeg8Kvideo12_157500.019120.079120.066010.02689
VLA-OSfold_560.013420.027910.039030.01565
VLA-OSrope_220.012410.019700.034390.01382
VLA-OSjean_770.014830.026570.028740.01546

R Q3:Comparison with Segmentation-Based Lookup and Semantic Conditioning

For generation, we evaluate semantic-map conditioning using the same Wan2.2 backbone (Table 9). Material-only outperforms Semantic-only across all metrics, while Depth+Material performs best overall.

Table 9. Conditioning ablation on Cholec80-5K.

BackboneConditionFVD ↓FID ↓SSIM ↑PSNR ↑LPIPS ↓Subj ↑Mot ↑Flicker ↑Bg ↑
Wan2.2Semantic60.6067.320.58417.310.36293.0098.8198.1796.60
Wan2.2Material56.7464.960.58617.380.35693.4798.8698.1896.79
Wan2.2D+M55.0966.890.58917.600.34893.7298.9898.3796.95

R Q4: Baseline Adaptation and Ablation on VLA-OS Dataset

For external generation baselines, we use their officially released checkpoints, which have already been trained on data from the target domain by the original authors. All methods are evaluated on the same test clips at matched resolutions using identical evaluation metrics to ensure a fair comparison. The checkpoint details and evaluation settings for each baseline are provided in Appendix D.2.

We further conduct conditioning ablations on VLA-OS, as shown in Table 8. Among the four configurations, our Depth+Material conditioning achieves the best overall performance across the reported metrics.

Table 8. Conditioning ablation on VLA-OS.

DMFVD ↓FID ↓SSIM ↑PSNR ↑LPIPS ↓Subj ↑Mot ↑Flicker ↑Bg ↑
××36.9339.850.846220.120.210793.4399.3399.0695.35
✓×35.8338.540.855820.740.193594.4299.4399.2895.86
×✓32.4036.090.857121.010.188794.6399.4299.3196.04
✓✓32.3134.950.857921.060.189494.9799.4499.2996.21

R Q5: Related Work

We added the suggested papers to the Related Work section.

Dear Reviewer 7Hxf:

Thank you again for taking the time to review our paper and share your feedback. We have provided additional experimental results and additional video results for the simulation demonstration on our project link in the abstract. We hope our responses have addressed your concerns.

As the discussion period comes to an end, we wanted to check if you have any further questions or comments. We would be happy to clarify anything that is still unclear.

Thank you for your time and thoughtful suggestions.

Best regards, The Authors

Reviewer_YQxROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PhysVerse connects monocular reconstruction, future video generation, and deformable simulation through a representation containing depth, Gaussian primitives, and dense maps of Young’s modulus EE, Poisson’s ratio nunu, and density rhorho.

The reconstruction component freezes HunyuanWorld-Mirror and trains a Physics DPT head to predict these material maps. For generation, the initial RGB frame, text, and reconstructed first-frame depth and material maps condition a frozen Wan2.2-TI2V-5B model through trainable control modules.

The clearest supported contribution is therefore an efficient pipeline that predicts material priors, uses them as generation conditions, and initializes a deformable simulator.

Critical strengths
  1. A useful problem and a coherent system interface.

  2. Efficient adaptation of pretrained models.

  3. The authors explicitly acknowledge category-level supervision, limited within-material variation, non-metric simulation calibration, and the restriction of feedback experiments to one qualitative reconstruction pass.

Critical weaknesses
  1. The central evaluation measures the recovery of semantic material priors, and the simple baseline is missing. The supervision design is central to the paper. Each semantic category receives a fixed database triplet. A predictor can obtain accurate material maps by recognizing semantic categories and reproducing their assigned constants. An additional experiment should evaluate mechanics independently of the annotation procedure.

  2. The simulator interface leaves consequential parts of the mechanical state unspecified. The paper does not sufficiently explain how VpV_p is determined, whether visible-surface Gaussians receive thickness or interior filling, or how Gaussians from different frames are fused without duplicating mass.

  3. The novelty is primarily integration, and the closest comparisons are incomplete. The authors appropriately cite several close predecessors; however, Tables 1–2 evaluate backbone choices and global material assignments rather than comparing this distinction against the closest material-estimation or physical-reconstruction approaches.

Questions for authors
  1. How does the Physics Head compare with a capacity-matched segmentation predictor followed by the exact cached material table? Can oracle segmentation establish the upper bound?

  2. What remains trainable in the no-depth/no-material ablation, and which external video baselines received equivalent adaptation and conditioning?

  3. How are particle volumes, rest states, rigid/soft contact, and dynamic Gaussian lifespans constructed? Can the exact configurations, material tables, and joint split manifests be provided?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YQxR

R W1/Q1: Segmentation-and-Lookup Baseline and Evaluation

We have added a baseline (Seg2Mat) that uses a LoRA-fine-tuned SAM model to predict semantic segmentation and assigns material properties to each class, reported in Table 1.

Unlike Seg2Mat, which maps each predicted class to a fixed material value and thus inherits any segmentation error, our Physics Head predicts dense material properties directly from visual features, without class-wise lookup. While Seg2Mat performs better on certain datasets, our Physics Head achieves lower errors on others without explicit semantic lookup, suggesting that direct material prediction is a viable alternative to class-wise assignment.

To further validate physical accuracy independently of semantic annotations, we evaluate against real physical measurements on PorcineEndo (Table 6). Our method achieves the lowest Poisson's ratio error and competitive results for other properties, demonstrating its potential beyond semantic-label matching.

Table 1. Material property prediction. Lower is better.

DatasetMethodALDE(E) ↓ADE(ν) ↓ARE(ρ) ↓DatasetMethodALDE(E) ↓ADE(ν) ↓ARE(ρ) ↓
CholecSeg8KPhysVerse (Ours)0.08360.00360.0534VLA-OSPhysVerse (Ours)0.08450.00430.0425
Seg2Mat0.17940.01060.1867Seg2Mat0.03190.00120.0070
EndoVis2018PhysVerse (Ours)0.07200.00350.0741VKittiPhysVerse (Ours)0.24870.01490.1557
Seg2Mat0.07900.00430.1269Seg2Mat0.34320.02190.1698
CaDISv2PhysVerse (Ours)0.09870.00660.0441UrbanSynPhysVerse (Ours)0.14460.00840.0761
Seg2Mat0.04700.00300.0488Seg2Mat0.34240.00930.0838

Table 6-1. Material prediction on the PorcineEndo dataset.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse (HWM)0.42790.007760.3232
PhysVerse (DA3)0.40720.009270.2727
PhysVerse (VGGT)0.42940.010420.3602

Table 6-2. Material prediction on the PorcineEndo dataset.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse (Ours)0.42790.007760.3232
PhysGM (Lv et al., 2026)0.60360.06810—
VoMP (Dagli et al., 2025)3.65510.116420.3929

R W2/Q3: MPM Simulation Setup

Since monocular reconstruction primarily captures the visible tissue surface, we add interior simulation particles behind the reconstructed surface to provide volumetric support. In normalized simulation coordinates, each MPM particle is assigned an effective volume of Vp=(gridlim/ngrid)3V_p=(\mathrm{grid_lim}/n_{\mathrm{grid}})^3. We initialize MPM from the Gaussians at a single frame rather than accumulating Gaussians across frames, so particle mass is not counted multiple times. We have clarified these details in Sections D.4 and E.4.

Our simulator supports rigid-soft contact, while the current quantitative experiments use a prescribed impulse for controlled evaluation. The time-varying Gaussians, including lifespan parameters, are predicted by the pretrained WorldMirror reconstruction head, which remains frozen. The specific simulation settings, material assignments, and dataset splits are documented in Appendix D.4,C.1, and D.3.

R W3: Comparison with Physical Reconstruction Methods

We additionally evaluate PhysVerse against PhysGM and VoMP on CholecSeg8K. PhysVerse achieves lower prediction errors than both methods across their respective comparable material properties.

Table 7. Material Prediction on CholecSeg8K. Lower is better. material property.

MethodALDE<sub>E</sub> ↓ADE<sub>ν</sub> ↓ARE<sub>ρ</sub> ↓
PhysVerse0.08360.00360.0534
PhysGM (Lv et al., 2026)0.60260.06678—
VoMP (Dagli et al., 2025)3.79660.070630.2684

R Q2 : Ablation Training and Baseline Adaptation

Across ablations, the backbone remains frozen, while the control branch remains trainable. For external video baselines, we use their officially released checkpoints, which have been trained on target-domain data by the original authors. All methods are evaluated under identical testing conditions. We additionally adapt Cosmos-H-Surgical with Depth+Material conditioning. Its official surgical-domain checkpoint improves across all reported metrics after conditioning.

BackboneConditionFVD ↓FID ↓SSIM ↑PSNR ↑LPIPS ↓Subj-Cons ↑Mot-Smooth ↑Temp-Flicker ↑Bg-Cons ↑
Cosmos-H-Surgical (zero-shot)None63.1369.930.54416.480.37293.3298.7898.1995.07
Cosmos-H-SurgicalDepth+Material57.4269.150.58817.270.34893.6899.2098.6096.68

Dear Reviewer YQxR:

Thank you again for taking the time to review our paper and share your feedback. We hope our responses have addressed your concerns.

As the discussion period comes to an end, we wanted to check if you have any further questions or comments. We would be happy to clarify anything that is still unclear.

Thank you for your time and thoughtful suggestions.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We have uploaded a revised paper that adds short clarifications and a few supplementary comparisons, shown in blue.

Key Reviewer Concerns and Responses

  • Physics Head vs. segmentation and lookup: We compare the Physics Head with Seg2Mat using material-prediction and simulation metrics, and analyze their prediction errors (Tables 1–2 and the reviewer response).

  • Evaluation against measured properties: We compare material predictions with physical-property measurements on PorcineEndo (Appendix A.3, Table 6).

  • Video-generation gains: We add conditioning ablations on VLA-OS and comparisons across generation backbones (Appendix A.5, Tables 8–9).

  • Comparison with related physical prediction methods: We compare material predictions with PhysGM and VoMP (Appendix A.4).

Detailed Manuscript Revisions

Related Works (Section 2)

  • We added and discussed the suggested papers to the Related Work section.

Methods (Section 3)

  • Appendix D clarifies the external video baseline settings, and MPM simulation setup (Sections D.2, D.5).
  • Appendix E.4 further details particle initialization and volume assignment.

Experments (Section 4)

  • Section 4.2.1 adds a segmentation-and-lookup baseline for material prediction and simulation (Tables 1–2; implementation details in Appendix D.4).
  • Appendix A.3 evaluates material predictions against real-world physical measurements (Table 6).
  • Appendix A.4 adds an material prediction comparison with related physical reconstruction methods (PhysGM and VoMP).
  • Appendix A.5 adds VLA-OS conditioning ablations and cross-backbone comparisons (Tables 8–9).
  • Appendix A.6 adds a novel-view rendering comparison (Table 10).

Supplementary/ Demo

We adds more simulation demo and results on https://anonymous.4open.science/w/doaneqzp-0DBD/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.