acceptodds
Under review as a conference paper at ICLR 2027

EfM: Simulation-Free Elastic Parameter Estimation from Observed Motion

Abstract

Reconstructing an object's 3D geometry and appearance captures how it looks, but its material properties are needed to simulate how it responds to new forces. Existing video-based material estimators either rely on costly inverse simulation or use feed-forward predictors whose material estimates are not explicitly constrained to satisfy the observed dynamics. We present EfM (Elasticity from Motion), which directly recovers the elastic parameters of freely moving objects from multi-view video without running a simulator. EfM makes this direct solve possible with two components: an F-GNN that infers unobserved volumetric deformation from visible surface motion, and a space-time weak formulation that avoids explicit acceleration estimation from observed surface trajectories. Together, they reduce material estimation to a single linear solve for Young's modulus and Poisson's ratio. On PAC-NeRF and Spring-Gaus-Diverse, EfM reduces Young's modulus error by 7.5 and 3.3 over the best baseline, respectively, predicts unseen future motion most accurately, and fits a sequence in 0.74s, three orders of magnitude faster than inverse simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
System Identification, Physics-based Simulation, 4D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_PcdWOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the EfM (Elasticity from Motion) method that focuses on estimating elasticity parameters of moving objects using a multi-view RGB observation setup. The experiments are conducted on PAC-NeRF and Spring-Gaus-Diverse, demonstrating an improvement over the baselines in terms of accuracy and efficiency.

Critical strengths
  • A timely method that focuses on leveraging physics and motion-governing paradigms instead of relying on exposing the learning systems to naive losses and hence data-hungry methods (and eventually computationally expensive simulators)
  • Well written and motivated piece of research, strong results, easy to follow and with a relevant Appendix
Critical weaknesses
  • More about the perception only. It is nudged in the paper that the setup assumption is a multi-view camera set. In my view, this also translates to the later statements in L. 298 (“, whether a camera sees it,” and “displacements over five consecutive frames”), which indicates a reliance on close-to-ideal observation to be able to infer the interior field prediction. Also, the fixed 5 frames seems a bit arbitrary, without much justification. What is the heuristic behind it? What if there are only two cameras with a big baseline? It seems that it is dictated by the specific experimental datasets rather than the design of the method. The general experimental setup of the camera placement is unclear (also in 4.6, it is mentioned that three cameras have been used, but what kind of coverage is provided?). It is hard to infer the robustness wrt the perception system used, e.g., would that work with a single view or 20% overlap two cameras? This is addressed to some degree in A3 and A4 but an important differentiator is the actual baseline setup. E.g., a camera observing an object from the back together with one from the front
  • This links up to the assumption of the assumed trajectory provided and as mentioned by the end of the paper (l 485) the error will grow dependent on its accuracy.
Questions for authors
  • Could you provide more details on the limits of the system wrt camera setup? Elaborate on the A3 and A4 connecting to my mentioned Weakness above. It would be great to provide an ablation with more specific views’ description (baselines, observation angle, maybe even a sketch if possible - and relation to the overall performance).
  • I was wondering about the chosen 3D representation. Why was the tetrahedral mesh (l. 152) chosen? Why not point clouds, voxels, Gaussians or any other representation? Is the watertightness an important aspect? I would welcome a discussion on that matter.
7 comments

We thank you for your valuable comments and for appreciating our timely, physics-based approach, its strong results and the clarity of the paper. We revised the manuscript accordingly (changes in blue) and added a supplementary results video. Below, we address the concerns raised in the review.


Weakness 1. "More about the perception only. It is nudged in the paper that the setup assumption is a multi-view camera set. In my view, this also translates to the later statements in L. 298 (", whether a camera sees it," and "displacements over five consecutive frames"), which indicates a reliance on close-to-ideal observation to be able to infer the interior field prediction. Also, the fixed 5 frames seems a bit arbitrary, without much justification. What is the heuristic behind it? What if there are only two cameras with a big baseline? ..."

Reply: Thank you for this insightful comment. Like the multi-view baselines we compare with, EfM works best when several cameras observe the object, and we mention this in the limitations (Sec. 5). Following the suggestion, we measured how much it actually needs. The F-GNN is trained with partial views from only 1 to 3 random directions with noise in half of its training steps (Sec. 3.2) and never sees the benchmark cameras. As a result, two cameras with a wide baseline already beat every baseline that uses all cameras (Table R4, App. C.5), whereas a single camera raises the error several times. As for the five input frames, they are the fewest symmetric frames that give velocity and acceleration at the center frame. Spacing them 2 or 3 frames apart still keeps EfM 2.2 to 5.6 times more accurate than the best baseline (Table R1, App. C.6).

Table R1. Spacing of the five input frames, without retraining. Best baseline: best value of any baseline in Table 2.

VariantPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrow
Paper (1 apart)0.0720.0330.2140.073
2 frames apart0.0970.0200.2500.070
3 frames apart0.1410.0220.3160.076
Best baseline0.5420.0720.6980.156

Weakness 2. "This links up to the assumption of the assumed trajectory provided and as mentioned by the end of the paper (l 485) the error will grow dependent on its accuracy."

Reply: Thank you for the comment. We now measure this dependence with tracked input. Since tracked trajectories are noisier than the given surface trajectory, we keep the same solve and replace the backbone of the F-GNN with a point-based Transformer trained on observations with simulated tracking error (App. C.3). On trajectories tracked from the benchmark videos, EfM then has the lowest E error of all methods (Table R2). On real captures, EfM's material also predicts unseen frames better than 10$\times$ stiffer or softer materials on every object (Table R3).

Table R2. EfM on trajectories tracked from the benchmark videos. Last row: EfM of Table 2 on O\mathcal{O}, for reference.

MethodPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowPAC CDrec\text{CD}_\text{rec} ↓\downarrowPAC CDpred\text{CD}_\text{pred} ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div CDrec\text{CD}_\text{rec} ↓\downarrowSG-Div CDpred\text{CD}_\text{pred} ↓\downarrow
EfM (ours)0.1370.0630.170.600.4030.1240.871.30
Vid2Sim I0.5640.1440.763.450.6980.1560.620.97
Vid2Sim I+II0.5420.1610.773.320.7330.1610.821.27
PhysGM0.8050.0721.163.281.0140.2311.543.61
Spring-Gaus--0.231.03--1.239.61
ReconPhys--1.106.40--2.109.87
EfM (O\mathcal{O} input, reference)0.0720.0330.150.200.2140.0730.200.38

Table R3. Real captures: silhouette IoU on the 12 observed (IoUrec\text{IoU}_\text{rec}) and 8 unseen (IoUpred\text{IoU}_\text{pred}) frames, with the material estimated from the first 12.

MaterialIoUrec\text{IoU}_\text{rec} ↑\uparrowIoUpred\text{IoU}_\text{pred} ↑\uparrow
EfM (ours)0.7710.504
10$\times$ stiffer0.6030.011
10$\times$ softer0.6750.066

Question 1. "Could you provide more details on the limits of the system wrt camera setup? Elaborate on the A3 and A4 connecting to my mentioned Weakness above. It would be great to provide an ablation with more specific views' description (baselines, observation angle, maybe even a sketch if possible - and relation to the overall performance)."

Reply: Thank you for the suggestion. We evaluated every single camera and every camera pair of the benchmark cameras, without selecting any (Table R4, App. C.5, camera sketch in Fig. 11). Two cameras suffice for EfM, and wider, less overlapping pairs work better. In particular, a front/back pair beats every baseline that uses all cameras. A single camera raises the error several times, which we mention in the limitations (Sec. 5). For the real captures (Sec. 4.6), the three cameras stand 89 to 131$^\circ$ apart around the table, about 0.7 m from the object (App. A.2, Fig. 11).

Table R4. Single cameras and camera pairs of the benchmark cameras, grouped by the angle between the two cameras.

CamerasOverlapPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrow
1 camera-0.3100.954
2 cams, 0-60$^\circ$≈\approx 0.50.1860.772
2 cams, 60-120$^\circ$≈\approx 0.30.1680.648
2 cams, 120-180$^\circ$0.10-0.140.1230.517
All cameras (paper)0.0720.214
Best baseline (all cameras)0.5420.698

Question 2. "I was wondering about the chosen 3D representation. Why was the tetrahedral mesh (l. 152) chosen? Why not point clouds, voxels, Gaussians or any other representation? Is the watertightness an important aspect? I would welcome a discussion on that matter."

Reply: Thank you for the question. We use a tetrahedral mesh because it provides the two things the solve needs, points with volumes and deformation gradients, and it gives the F-GNN interior nodes to pass information through. Watertightness therefore matters only for placing these interior points. The solve itself only needs points that carry a volume and a deformation gradient, so EfM also runs on point clouds. For tracked trajectories, we replace the backbone of the F-GNN with a point-based Transformer (App. C.3), which runs on points alone and still gives the lowest E error of all methods on such input (Sec. 4.7, Table 4). We added this discussion to Sec. 3.

Reviewer_PcdWOpenReview ↗

Comprehensive clarifications

I thank the authors for providing a comprehensive answer to my raised questions.

Q1/A1 -- Excellent, this sheds more light on the limitations and required camera setup. I believe this is an insightful addition for a future adoption of the proposed solution.

Q2/A2 -- I see, this explains the design; and the revised discussion in the main paper clarifies and justfies the design as well. It is reasonalbe to use point transformer, I agree.

Thank you for engaging in the discussion.

Reviewer_1cfsOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to obtain 3D geometry, appearance, and material parameters from videos and images more quickly and accurately. Existing approaches either rely on many iterations of forward and backward simulation passes, which is computationally heavy, or on end-to-end data-driven feed-forward networks, which may not be accurate. Instead, the paper proposes equation-based material estimation, together with a network trained to provide the information needed to solve the equations.

The proposed method first turns material parameter estimation into a linear problem. However, some variables in this linear problem cannot be obtained accurately from images and videos. To address this, the paper uses a space-time weak form, which avoids directly estimating highly noisy and unstable quantities. In addition, to estimate the deformation gradient F, an important variable in this problem, the paper proposes a Graph Neural Network (F-GNN).

The experiments show that the method achieves much more accurate parameters and is much faster, in both material parameter estimation and future prediction.

Critical strengths

The method is well grounded mathematically and theoretically, and it shows significant improvements in physical parameter estimation and future prediction. Its equation-based approach would be helpful for researchers working on estimating physical parameters from images and videos.

Critical weaknesses

The experiments were done only on simple geometry, such as the objects in the PAC-NeRF and Spring-Gaus datasets. That is, experiments on more complex geometry are lacking. Since the performance seems to depend heavily on the accuracy of the F-GNN and of tracking each node, it would be important to show that the method also works on more complex geometry. As the paper already compares with Vid2Sim, the method could be tested relatively easily on the Vid2Sim dataset, which contains much more complex geometry.

Questions for authors
  • Would it be possible to show whether the method also works on more complex geometry, for example by testing on the Vid2Sim dataset?

  • The paper claims that the initial velocity and other unknowns could be easily included as additional unknowns in the linear solve. If so, would it be possible to explain how the initial velocity can be estimated in this way, and how accurately? Or would additional techniques be needed to overcome noise?

  • The paper states that "All methods start from the same frame-0 geometry and initial velocity." Could you specify how the frame-0 geometry is obtained?

5 comments

We thank you for your valuable comments and for appreciating the theoretical grounding of EfM and its significant improvements at a much lower cost. We revised the manuscript accordingly (changes in blue) and added a supplementary results video. Below, we address the concerns raised in the review.


Weakness 1. "The experiments were done only on simple geometry, such as the objects in the PAC-NeRF and Spring-Gaus datasets. That is, experiments on more complex geometry are lacking. Since the performance seems to depend heavily on the accuracy of the F-GNN and of tracking each node, it would be important to show that the method also works on more complex geometry. As the paper already compares with Vid2Sim, the method could be tested relatively easily on the Vid2Sim dataset, which contains much more complex geometry."

Reply: Thank you for the suggestion. We ran EfM on complex objects from the Vid2Sim test set (Google Scanned Objects), which have thin parts, holes and concave shapes, reading the material from its rendered videos (Table R1, App. C.2, Fig. 9). Vid2Sim generates its data with finite elements, so we assemble the solve and re-simulate the estimate on finite elements, as Sec. 3.1.2 (L264-269) does for the simulator that uses the estimate. On the blocks and the bus, EfM recovers E with ∣log⁡E∣\lvert\log E\rvert errors of 0.010 and 0.089 and ν\nu with ∣Δν∣\lvert\Delta\nu\rvert errors of 0.038 and 0.052, and its re-simulation follows the rendered fall (Table R1, supplementary video 1:12).

Table R1. Complex geometry (GSO dataset) from the Vid2Sim test renders: EfM on tracked trajectories, re-simulated with finite elements. Silhouette IoU over 12 views on the 16 observed (IoUrec\text{IoU}_\text{rec}) and 8 unseen (IoUpred\text{IoU}_\text{pred}) frames.

Object∣log⁡E∣\lvert\log E\rvert ↓\downarrow∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowIoUrec\text{IoU}_\text{rec} ↑\uparrowIoUpred\text{IoU}_\text{pred} ↑\uparrow
blocks0.0100.0380.9100.908
bus0.0890.0520.8780.853

Question 1. "The paper claims that the initial velocity and other unknowns could be easily included as additional unknowns in the linear solve. If so, would it be possible to explain how the initial velocity can be estimated in this way, and how accurately? Or would additional techniques be needed to overcome noise?"

Reply: Thank you for the question. EfM can estimate the initial velocity in the same linear solve, in the same way as the unknown ground contact force (Sec. 3.1.2). A time window that starts at t = 0 adds a term that is linear in the initial velocity, so the initial translation and rotation become six extra unknowns (derivation in App. C.9). On the PAC-NeRF benchmark, the estimate is within about 2% of the object speed, close to the 1.2% that GIC reaches with a differentiable simulator but thousands of times faster (Table R2). The joint estimate leaves (E,ν)(E, \nu) unchanged, and re-simulating with the estimated initial velocity gives almost the same CD as with the true one (CDrec\text{CD}_\text{rec} 0.146 vs. 0.147, CDpred\text{CD}_\text{pred} 0.201 vs. 0.202).

Table R2. Initial velocity on the PAC-NeRF benchmark: error relative to the object speed and time per sequence.

MethodHow v(0)v(0) is obtainedv0v_0 error ↓\downarrowTime ↓\downarrow
GICAdam through differentiable MPM1.2%1115 s
EfM (ours)Six columns of the same solve2.3%0.07 s

Question 2. "The paper states that "All methods start from the same frame-0 geometry and initial velocity." Could you specify how the frame-0 geometry is obtained?"

Reply: Thank you for the question. Every method receives the simulator's initial point set as the frame-0 geometry, which keeps geometry errors out of the material comparison, and we now state this in Sec. 3 and App. B.7. Even with a frame-0 geometry reconstructed from the videos, EfM is equally accurate on the PAC-NeRF benchmark (Table R3, App. C.4).

Table R3. Given vs. reconstructed frame-0 geometry on the PAC-NeRF benchmark.

Frame 0PAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrow
Given0.0720.033
Reconstructed0.0700.032

Thank you so much for your detailed comments. Most of my concerns have been addressed, particularly those related to Q1 and Q2.

Not critical, but, for the complex objects, I still feel that the complex geometry examples are not entirely convincing. The "blocks" and "bus" examples are relatively simple objects in the GSO datasets. They have less convex and thin structures, where small local details can be critical. A clearer demonstration of how object complexity impacts performance would help us determine whether this is a limitation or a genuine strength of the proposed method.

Thank you for the follow-up and for engaging so actively in the discussion. We are glad that our responses addressed most of your concerns, particularly Q1 and Q2. To address the remaining one, we built shapes of increasing complexity by splitting branches dd times, which makes them less convex with thin structures (App. D, Fig. 15). Each shape is dropped with a soft and a stiffer material, and the videos of all sequences are in the updated supplementary material (efm_complexgeom_sequence_video.mp4).

As Table R4 shows, our solve stays accurate at every level of complexity. Its first row runs the space-time weak-form solve (Sec. 3.1) with the simulator's deformation gradient in place of the F-GNN output (Sec. 3.2), so it measures the solve alone. When the F-GNN predicts the interior from the visible surface, the error grows on deeply branched shapes, because its original training set contains no such complex shapes (Fig. 16a). A short fine-tuning on other branched shapes (Fig. 16b), disjoint from the tested ones, reduces this error, most at the deepest level. Importantly, the fine-tuned F-GNN keeps its accuracy on the PAC-NeRF and Spring-Gaus-Diverse benchmarks (Table R5), so the gain does not come at the cost of other shapes. Complex shapes are therefore a current limitation of the learned F-GNN, not of the solve, and one that broader training data can address, which we added to the limitations (Sec. 5).

Table R4. ∣log⁡E∣\lvert\log E\rvert on branching shapes (mean over three shapes per level and the two materials).

dd01234
Solve with true F\mathbf{F}0.0040.0130.0080.0240.027
EfM (F-GNN)0.0510.0270.2190.1640.570
EfM (F-GNN, fine-tuned)0.1000.0250.1890.1110.125

Table R5. EfM on the PAC-NeRF and Spring-Gaus-Diverse benchmarks with the original F-GNN of the paper and the fine-tuned F-GNN.

F-GNNPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrow
Original0.0720.0330.2140.073
Fine-tuned0.0500.0370.2120.079
Reviewer_UPw7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a simulation-free method to estimate Young's modulus and Poisson’s ratio of the object from multi-view videos. This method greatly reduces the time required to estimate the parameters compared to other baselines. The method combines a space-time weak formulation that avoids explicit acceleration estimation with an F-GNN that predicts interior deformation gradients.

Critical strengths
  • The proposed method combines a linear solve for material parameters with learned interior deformation gradients.
  • The method substantially reduces fitting time while achieving strong material estimation and future-motion prediction accuracy on the evaluated benchmarks.
Critical weaknesses
  • On synthetic data, EfM receives ground-truth visible surface trajectories, whereas the baselines consume videos. The appendix adapts Vid2Sim Stage II to use the same trajectories, but its Stage I predictor was not trained for this input. A more direct comparison would evaluate EfM using trajectories estimated from the same videos, as in the real-data experiments. This would also reveal its robustness to tracking errors.
  • The paper reports quantitative material estimation results on Spring-Gaus-Diverse, but only qualitative reconstruction results on the original Spring-Gaus dataset. The authors might also include the quantitative results for future predictions on the original dataset.
  • The authors could also include GIC or MASIV as MPM-based inverse simulation baselines. Most existing baselines use different simulation frameworks, while PhysGM uses MPM but does not perform per-scene inverse simulation.
Questions for authors
  • How is the initial velocity obtained in the experiments? Is it provided as ground truth, assumed to be zero, or estimated from observations? Can the method handle unknown initial velocities or jointly estimate them with the material parameters?
  • Other questions are already asked in the critical weaknesses.
6 comments

We thank you for your valuable comments and for appreciating how EfM combines a linear solve with learned interior deformation gradients and greatly reduces the fitting time with strong accuracy. We revised the manuscript accordingly (changes in blue) and added a supplementary results video. Below, we address the concerns raised in the review.


Weakness 1. "On synthetic data, EfM receives ground-truth visible surface trajectories, whereas the baselines consume videos. The appendix adapts Vid2Sim Stage II to use the same trajectories, but its Stage I predictor was not trained for this input. A more direct comparison would evaluate EfM using trajectories estimated from the same videos, as in the real-data experiments. This would also reveal its robustness to tracking errors."

Reply: Thank you for the valuable suggestion. We now run EfM on trajectories tracked from multi-view videos of the same benchmark sequences (Table R1). Tracked trajectories are noisier than the given surface trajectory, and this tracking error adds apparent strain to the solve. We therefore keep the same solve and replace the backbone of the F-GNN, a graph network on the tetrahedral mesh, with a point-based Transformer trained on observations with simulated tracking error, which predicts the interior deformation gradient and also corrects the tracked positions (details in App. C.3). EfM has the lowest E error on both benchmarks, 4.0 and 1.7 times lower than the best baseline. We also added this comparison to the main text (Sec. 4.7, Table 4).

Table R1. EfM on trajectories tracked from the benchmark videos. Last row: EfM of Table 2 on O\mathcal{O}, for reference.

MethodPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowPAC CDrec\text{CD}_\text{rec} ↓\downarrowPAC CDpred\text{CD}_\text{pred} ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div CDrec\text{CD}_\text{rec} ↓\downarrowSG-Div CDpred\text{CD}_\text{pred} ↓\downarrow
EfM (ours)0.1370.0630.170.600.4030.1240.871.30
Vid2Sim I0.5640.1440.763.450.6980.1560.620.97
Vid2Sim I+II0.5420.1610.773.320.7330.1610.821.27
PhysGM0.8050.0721.163.281.0140.2311.543.61
Spring-Gaus--0.231.03--1.239.61
ReconPhys--1.106.40--2.109.87
EfM (O\mathcal{O} input, reference)0.0720.0330.150.200.2140.0730.200.38

Weakness 2. "The paper reports quantitative material estimation results on Spring-Gaus-Diverse, but only qualitative reconstruction results on the original Spring-Gaus dataset. The authors might also include the quantitative results for future predictions on the original dataset."

Reply: Thank you for the suggestion. As noted in Sec. 4.1, the released benchmark was designed for reconstruction rather than material estimation, and its materials span only one decade in E, which is too narrow to rank methods by the mean error (App. A.5). We therefore built Spring-Gaus-Diverse with the same objects and a wider range. Following the suggestion, we now also provide quantitative results on the released benchmark (Table R2, App. A.5). There, EfM is the only method whose estimates rise with the true E (Spearman +0.47), and it has the best ∣Δν∣\lvert\Delta\nu\rvert and the lowest median CDpred\text{CD}_\text{pred}. Two of the 14 sequences, the letter C and the toothpaste, stand upright and topple after landing, so a small error in EfM's material decides the direction of the fall and grows into a large drift (CDpred\text{CD}_\text{pred} 13.5 and 67.8). These two dominate EfM's mean CDpred\text{CD}_\text{pred}, which is 2.24 without them, second to that of Vid2Sim Stage I (1.99). We show both cases in the limitations part of the supplementary video (3:33) and in Sec. 5.

Table R2. Released Spring-Gaus benchmark, all 14 sequences. The fifth column gives the mean CDpred\text{CD}_\text{pred} over the 12 sequences other than the two that topple after landing (C and toothpaste). Spearman: rank correlation of estimated and true E.

Method∣log⁡E∣\lvert\log E\rvert ↓\downarrow∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowCDpred\text{CD}_\text{pred} mean ↓\downarrowCDpred\text{CD}_\text{pred} mean, w/o C and toothpaste ↓\downarrowCDpred\text{CD}_\text{pred} median ↓\downarrowSpearman ↑\uparrow
EfM (ours)0.3740.0917.732.240.64+0.47
Vid2Sim I0.2990.1172.481.991.26$-$0.29
Vid2Sim I+II0.3560.1242.852.381.19$-$0.17
PhysGM0.5210.1573.942.652.08$-$0.38
Spring-Gaus--4.804.241.98-
ReconPhys--5.473.392.40-

Weakness 3. "The authors could also include GIC or MASIV as MPM-based inverse simulation baselines. Most existing baselines use different simulation frameworks, while PhysGM uses MPM but does not perform per-scene inverse simulation."

Reply: Thank you for the suggestion. Although Table 2 compares with Vid2Sim, the more recent inverse-simulation method, which uses another simulator, we agree that MPM-based inverse simulation is also an important reference, so we now provide results for PAC-NeRF, GIC and MASIV under the protocol of Table 2 (Table R3, App. C.1). On Spring-Gaus-Diverse, EfM outperforms PAC-NeRF and GIC on every metric. On the PAC-NeRF benchmark, PAC-NeRF and GIC estimate E more accurately, but at this level of accuracy all three replay the observed and future motion almost exactly (CDrec\text{CD}_\text{rec} 0.14 to 0.15, CDpred\text{CD}_\text{pred} 0.15 to 0.20), so this benchmark is close to saturated, and the rollouts of EfM and PAC-NeRF are hard to tell apart in the supplementary video (0:43). EfM reaches this accuracy in under a second, about 7,000 to 10,000 times faster than PAC-NeRF and GIC. MASIV learns a neural constitutive law and returns no (E,ν)(E, \nu), so we compare it on motion only. Its released schedule of 1000 epochs takes one to two days per sequence on our GPU, so we fitted it for 100 epochs (2.5 to 3.4 h per sequence), already longer than PAC-NeRF and GIC take. Its fit of the observed frames converges within this budget, but its future frames drift, and 4 of the 10 PAC-NeRF rollouts diverge.

Table R3. MPM-based inverse simulation under the protocol of Table 2. ∗^*Mean over the 6 of 10 rollouts that stay bounded.

MethodPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowPAC CDrec\text{CD}_\text{rec} ↓\downarrowPAC CDpred\text{CD}_\text{pred} ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div CDrec\text{CD}_\text{rec} ↓\downarrowSG-Div CDpred\text{CD}_\text{pred} ↓\downarrowTime ↓\downarrow
EfM (ours)0.0720.0330.150.200.2140.0730.200.380.74 s
PAC-NeRF0.0300.0440.140.150.2730.1740.300.601.5-2 h
GIC0.0240.0250.140.150.2440.2270.211.041.4 h
MASIV--0.433.26$^*$--0.6116.02.5-3.4 h

Question 1. "How is the initial velocity obtained in the experiments? Is it provided as ground truth, assumed to be zero, or estimated from observations? Can the method handle unknown initial velocities or jointly estimate them with the material parameters?"

Reply: Thank you for the question. All methods receive the benchmark's true initial velocity (Sec. 4.1, App. B.7), which keeps errors in the initial state out of the material comparison. All methods use it to re-simulate their estimates. The baselines that fit a simulator to the video also start their fitting from it instead of estimating it themselves, while EfM does not need it to estimate the material.

EfM can also estimate the initial velocity jointly with the material, in the same linear solve. A time window that starts at t = 0 adds a term that is linear in the initial velocity, so the initial translation and rotation become six extra unknowns (derivation in App. C.9). On the PAC-NeRF benchmark, the estimate is within about 2% of the object speed, close to the 1.2% that GIC reaches with a differentiable simulator but thousands of times faster (Table R4). The joint estimate leaves (E,ν)(E, \nu) unchanged, and re-simulating with the estimated initial velocity gives almost the same CD as with the true one (CDrec\text{CD}_\text{rec} 0.146 vs. 0.147, CDpred\text{CD}_\text{pred} 0.201 vs. 0.202).

Table R4. Initial velocity on the PAC-NeRF benchmark: error relative to the object speed and time per sequence.

MethodHow v(0)v(0) is obtainedv0v_0 error ↓\downarrowTime ↓\downarrow
GICAdam through differentiable MPM1.2%1115 s
EfM (ours)Six columns of the same solve2.3%0.07 s
Reviewer_bVW1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to estimate properties of elastic objects from multiview videos. The method can recover parameters through a direct physics-based solve of the governing equation. The paper introduces F-GNN, a graph neural network that predicts the deformation gradient, which is then used to compute the governing equation and parameters. The method further uses a space-time weak formulation to avoid differentiating through noisy trajectories. Results show improved performance over baselines.

Critical strengths
  • A clean formulation that does seem like an improvement. The method is carefully thought of, avoiding errors I would have expected around noise.
  • The paper is very well written
Critical weaknesses
  • No video results. This is a major weakness to me as the numbers do not do a full justice to how the simulations appear
  • No comparisons with PAC-NeRF. Differentiable simulator-based approaches like PAC-NeRF or PhysTwin also use governing equations to fit to observations. Why does the paper not compare to them?
  • Synthetic experiments use GT trajectories, which does not provide a natural picture of real-world performance. Real-world evaluation does not evaluate future evolution or new interventions, so this is important.
  • The limitations around homogeneous materials is shared with other methods, but I do not understand how the method could potentially be extended to more diverse objects.
Questions for authors

See weaknesses

4 comments

We thank you for your valuable comments and for appreciating our clean formulation and the clarity of the paper. We revised the manuscript accordingly (changes in blue) and added a supplementary results video, which addresses your main concern. Below, we address the concerns raised in the review.


Weakness 1. "No video results. This is a major weakness to me as the numbers do not do a full justice to how the simulations appear"

Reply: Thank you for the comment. We agree, and we added a results video to the supplementary material. It shows the qualitative comparisons of the paper and the new experiments of this response, and the revised manuscript shows the same comparisons as figures (Fig. 4, 8 to 10 and 12 to 14).


Weakness 2. "No comparisons with PAC-NeRF. Differentiable simulator-based approaches like PAC-NeRF or PhysTwin also use governing equations to fit to observations. Why does the paper not compare to them?"

Reply: Thank you for the comment. We would like to note that our comparison already includes two differentiable-simulation methods that are more recent than PAC-NeRF, Vid2Sim (CVPR 2025) and Spring-Gaus (ECCV 2024), in Table 2. Like PAC-NeRF (ICLR 2023), they fit a simulator to the video over many iterations, whereas EfM solves for the material once. PhysTwin addresses a different task, the manipulation of an object by hand, so we did not include it in the original comparison. That said, following the suggestion, we now also compare with PAC-NeRF and PhysTwin below.

PAC-NeRF (Table R1, App. C.1, Fig. 8). We ran PAC-NeRF under the protocol of Table 2. On Spring-Gaus-Diverse, EfM is better on every metric and thousands of times faster. On the PAC-NeRF benchmark, PAC-NeRF estimates E more accurately and EfM estimates ν\nu more accurately, yet both replay the observed and future motion almost equally well (CD 0.15 vs. 0.14 and 0.20 vs. 0.15), as the supplementary video shows (0:43).

Table R1. PAC-NeRF under the protocol of Table 2

MethodPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowPAC CDrec\text{CD}_\text{rec} ↓\downarrowPAC CDpred\text{CD}_\text{pred} ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div CDrec\text{CD}_\text{rec} ↓\downarrowSG-Div CDpred\text{CD}_\text{pred} ↓\downarrowTime ↓\downarrow
EfM (ours)0.0720.0330.150.200.2140.0730.200.380.74 s
PAC-NeRF0.0300.0440.140.150.2730.1740.300.601.5-2 h

PhysTwin (Table R2, App. C.8, Fig. 13, supp. video 2:44). PhysTwin builds a digital twin of an object that a person deforms by hand, so the hands apply forces that change over time and must be modeled, whereas EfM and the baselines of Table 2 read the material of an object that falls freely and hits the ground. Following R-bVW1's suggestion, we adapted EfM to this task by treating the unknown forces of the tracked hands like the ground contact, as unknowns of the same solve, and ran it on three manipulation sequences of PhysTwin. EfM reads one homogeneous material from the tracked videos, as Vid2Sim, PhysGM, PAC-NeRF and GIC also do, and we roll it out in PhysTwin's simulator. EfM performs on par with PhysTwin on the dinosaur and the zebra, and it reads the material in seconds instead of 12 to 26 minutes. On the sloth, where two hands deform the object strongly, PhysTwin is more accurate, since it fits every spring of the object, whereas EfM reads a single material with an F-GNN that has never seen manipulation. We added this case to the limitations (Sec. 5).

Table R2. PhysTwin sequences: future-frame errors and time to fit the material.

SequencePhysTwin CD ↓\downarrowEfM CD ↓\downarrowPhysTwin Track ↓\downarrowEfM Track ↓\downarrowPhysTwin Time ↓\downarrowEfM Time ↓\downarrow
dinosaur, one-hand lift0.01250.01180.01470.014826 min5.7 s
zebra, two-hand lift0.01530.01570.03120.028012 min2.3 s
sloth, two-hand lift0.01960.02580.02810.035320 min9.9 s

Weakness 3. "Synthetic experiments use GT trajectories, which does not provide a natural picture of real-world performance. Real-world evaluation does not evaluate future evolution or new interventions, so this is important."

Reply: Thank you for the valuable comment. We would like to clarify that the synthetic benchmarks give EfM the surface trajectory to isolate material estimation from tracking errors. To show how EfM performs without this simplification, we now add three evaluations.

Tracked trajectories (Table R3, App. C.3, Fig. 10, supp. video 1:29). We now run EfM on trajectories tracked from the same benchmark videos that the baselines read. Tracked trajectories are noisier than the given surface trajectory and than the noise the F-GNN is trained with, and this tracking error adds apparent strain to the solve. We therefore keep the same solve and replace the backbone of the F-GNN, a graph network on the tetrahedral mesh, with a point-based Transformer. It is trained on observations with simulated tracking error, and it predicts the interior deformation gradient and also corrects the tracked positions (details in App. C.3). Even from tracked trajectories, EfM has the lowest E error on both benchmarks, 4.0 and 1.7 times lower than the best baseline.

Table R3. EfM on trajectories tracked from the benchmark videos. Last row: EfM of Table 2 on O\mathcal{O}, for reference.

MethodPAC ∣log⁡E∣\lvert\log E\rvert ↓\downarrowPAC ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowPAC CDrec\text{CD}_\text{rec} ↓\downarrowPAC CDpred\text{CD}_\text{pred} ↓\downarrowSG-Div ∣log⁡E∣\lvert\log E\rvert ↓\downarrowSG-Div ∣Δν∣\lvert\Delta\nu\rvert ↓\downarrowSG-Div CDrec\text{CD}_\text{rec} ↓\downarrowSG-Div CDpred\text{CD}_\text{pred} ↓\downarrow
EfM (ours)0.1370.0630.170.600.4030.1240.871.30
Vid2Sim I0.5640.1440.763.450.6980.1560.620.97
Vid2Sim I+II0.5420.1610.773.320.7330.1610.821.27
PhysGM0.8050.0721.163.281.0140.2311.543.61
Spring-Gaus--0.231.03--1.239.61
ReconPhys--1.106.40--2.109.87
EfM (O\mathcal{O} input, reference)0.0720.0330.150.200.2140.0730.200.38

Future evolution on real captures (Table R4, Fig. 4, App. A.2, supp. video 2:28). EfM now estimates the material from only the first 12 of the 20 frames, and as in Sec. 4.6, we compare its re-simulation with 10$\times$ stiffer and 10$\times$ softer materials. EfM's material predicts the 8 unseen frames best on all five objects, with a mean IoU of 0.50 against 0.01 and 0.07. We revised Sec. 4.6 and Fig. 4 accordingly.

Table R4. Real captures: silhouette IoU on the 12 observed (IoUrec\text{IoU}_\text{rec}) and 8 unseen (IoUpred\text{IoU}_\text{pred}) frames, with the material estimated from the first 12.

MaterialIoUrec\text{IoU}_\text{rec} ↑\uparrowIoUpred\text{IoU}_\text{pred} ↑\uparrow
EfM (ours)0.7710.504
10$\times$ stiffer0.6030.011
10$\times$ softer0.6750.066

New interventions (Table R5, App. C.7, Fig. 12, supp. video 1:52). We re-simulated every benchmark object with each method's estimate under initial conditions never observed, a sideways toss and a strong spin. EfM's estimate reproduces the true motion with 2 to 7 times lower CD than the estimates of the baselines.

Table R5. Unseen initial conditions: CD to the ground-truth motion under the new initial conditions.

MethodPAC Toss ↓\downarrowPAC Spin ↓\downarrowSG-Div Toss ↓\downarrowSG-Div Spin ↓\downarrow
EfM (ours)0.1580.1520.2550.419
Vid2Sim I1.0081.1200.7311.026
Vid2Sim I+II1.0041.0430.8811.111
PhysGM1.2902.5912.3001.626

Weakness 4. "The limitations around homogeneous materials is shared with other methods, but I do not understand how the method could potentially be extended to more diverse objects."

Reply: Thank you for the question. To clarify, we now describe the extension in detail and provide experimental results (App. C.10). EfM extends to heterogeneous materials without changing the solve. The stress is linear in the material parameters at every point, so we let the stiffness vary in space as a smooth field with a few unknowns, a triplane as in PhysDreamer (Zhang et al., ECCV 2024). Each unknown then adds one column to the same linear system, and one solve recovers the whole field. We evaluated this extension on 9 objects with spatially varying stiffness. The recovered stiffness map follows the true one with a Pearson correlation of 0.82, and the E error drops from 0.41 with a single E to 0.12 (Table R6, Fig. 14, supp. video 3:02). This test uses the simulator's deformation gradient to isolate the solve, so training the F-GNN on heterogeneous objects is the next step.

Table R6. Material estimation performance of 9 heterogeneous objects. ∣log⁡E∣\lvert\log E\rvert is the median error over the points of each object

Material fieldPearson ↑\uparrowSpearman ↑\uparrow∣log⁡E∣\lvert\log E\rvert ↓\downarrow
Single E--0.41
Triplane 16$^2$0.820.860.12

Thank you again for your time and effort in reviewing our paper. As the reviewer-author discussion period ends soon, we would be grateful if you could let us know whether our responses have addressed your concerns, in particular the lack of video results, for which we added a results video to the supplementary material. If any points remain unclear, we would appreciate your further feedback and would be glad to clarify them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.