- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a feed-forward framework that infers physical material properties directly from monocular videos while preventing models from relying on geometric or appearance shortcuts. The approach uses a two-stage distillation process where a teacher model learns robust dynamics from simulated physical states, and a video-based student aligns its feature representations with that frozen teacher. This physics-grounded guidance enables the system to generalize effectively to unseen object shapes, maintain high inference efficiency, and transfer to real-world scenarios.
Critical strengths
-
The problem is well-motivated problem and the approach is intuitive. The paper identifies the challenging problem of separating physical dynamics from geometry and viewpoint, and proposed physical states as a supervision.
-
The separation between trajectory-based representation learning and video-based inference is convincing. The method transfers knowledge from a teacher that has access to simulation data to a student that operates on real-world data.
-
I appreciate the ablation studies, e.g., the comparison between the model without alignment, the alignment-based variant, the benefit of representation distillation, etc..
-
The method is also evaluated on unseen geometries and shorter input sequences, which supports the generalization of the method
-
The method is efficient, e.g., 3.8ms inference time per instance.
Critical weaknesses
-
The benchmark covers only 3 pre-defined material families, so it does not fully support the broad generalization claims and open-world problem setting. Furthermore, the simulations rely on a controlled environment with fixed settings, so it does not establish robustness to diverse physical interactions or real-world environmental conditions.
-
Similarly, the real-world evaluation only contains 2 scenes (Bun, Burger), so it does not fully support robust real-world generalization claims.
-
The evidence of physical grounding mainly relies on qualitative UMAP visualizations, so it does not fully serves as the evidence that the representations are physically invariant or disentangled from geometry. Overall, I think the paper might be overclaiming in multiple places.
-
The pre-processing latency is not included. If the pre-processing is slow, the 3.8ms latency claim does not include the bottleneck of the system.
Questions for authors
-
Does the method work beyond pre-defined material families (outside jelly, plasticine, and sand)?
-
Does the method work in real-world beyond Bun and Burger? Perhaps yes, but I would like to see more results.
-
Could the authors share the end-to-end latency with the breakdown to other components, e.g., pre-processing (not just 3.8ms inference time)?
3 comments
Response To Reviewer NGCn — Part 2
Table 1: Quantitative comparison of physical invariance to geometric variations and sensitivity to material parameters. geometry distance measures representation differences between shapes with the same material parameters (lower is better), while physics distance measures representation differences between parameter settings for the same shape (higher is better).
| Method | Geometry distance ↓ | Physics distance ↑ |
|---|---|---|
| Vid2Sim | 0.0351 | 0.0379 |
| Ours (w/o alignment) | 0.0666 | 0.0428 |
| Ours (alignment only) | 0.0154 | 0.1197 |
| Ours (Full) | 0.0146 | 0.1202 |
- For Weakness 3: Physical Invariance and Geometry Disentanglement
To assess whether the learned representations are organized in a physically meaningful manner, we conduct a comparative analysis of learned representations by different methods. We refer the reviewer to Appendix D.1 of the revised manuscript, line 828.
Specifically, for each material, we simulate all combinations of 10 parameter levels and 10 object shapes, yielding 100 trajectories. We vary Young's modulus for Jelly and Plasticine, and friction angle for Sand, while keeping all other simulation settings unchanged. Let (z_{p,s}) denote the dynamics latent extracted from the trajectory corresponding to parameter level (p) and shape (s).
We measure two complementary properties: geometry distance, measuring representation differences between different shapes with the same material parameters (lower values indicate greater physical invariance to geometric variations), and physics distance, measuring representation differences between different material parameters under the same shapes (higher values indicate better parameter discrimination):
D_{\mathrm{geo}} &= \frac{1}{P\binom{S}{2}} \sum_{p=1}^{P}\sum_{s<t} d(z_{p,s},z_{p,t}),\\ D_{\mathrm{phys}} &= \frac{1}{S\binom{P}{2}} \sum_{s=1}^{S}\sum_{p<q} d(z_{p,s},z_{q,s}),\\ d(z,z') &= 1-\cos(z,z'). \end{aligned}$$ where (P=S=10). As shown in **Table 1 above**, **Ours (Full) consistently achieves the lowest geometry distances and highest physics distances.** This indicates that our method learns representations with greater physical invariance to geometric variations and better discrimination between different material parameter settings. **Fig. 8 of the revised manuscript, line 773**, further reveals the parameter-wise similarity structure. **For Ours (Full), representations of nearby parameter levels exhibit higher similarity, which generally decreases as the parameter gap increases.** This graded structure is less pronounced in Ours (w/o alignment) and Vid2Sim, indicating that our learned representations better preserve the continuous variations associated with material parameters. **Revisions in paper:** We have **narrowed our claims** and highlighted the corresponding changes in blue at Lines 27–30, 87–89, 104–107, and 523–525. In addition, we have added the **More Analysis of Physics-Grounded Representations** experiments in Appendix D.1, lines 828–855. * **For Weakness 4 and Question 3: End-to-End Latency and Runtime Breakdown** Thank you for raising this important point. In real-world evaluation, we measure the processing latency from the captured RGB video to the final material prediction. The pipeline consists of two stages: - **Mask Extraction:** SAM2 takes **4,625 ms** to extract object masks. - **Model Inference:** Our Student network takes **3.83 ms** for material prediction. The resulting end-to-end latency is **4,628.83 ms (4.63 s)**. Previously, we reported only network inference latency because masks are directly available in our synthetic benchmark, without requiring additional extraction. **Revisions in paper:** We have **clarified in paper that the reported 3.83 ms refers to the model forward-pass time** (lines 27-30, lines 104-107, and lines 523-525), and provided a detailed breakdown of the end-to-end latency from RGB video input to material prediction in Appendix D.8, lines 1069–1075.I appreciate the analyses on sensitivity to material parameters, invariance to object geometry, latency, etc. Authors also provided results on four additional real-world objects, but I still have concerns about the model's ability to recognize arbitrary unseen material families. Nevertheless, the rebuttal reasonably addresses most of my concerns, so I raised my rating to weak accept.
Response To Reviewer NGCn — Part 1
Response To Reviewer NGCn:
Thank you for your insightful comments, which are greatly valued by us. We will address your concerns below.
We thank the reviewer for this valuable comment.
Generalization Beyond Pre-defined Material Families.
We select Jelly, Plasticine, and Sand as three representative material families for evaluating our method, rather than aiming to cover the full diversity of real-world materials. Importantly, our framework is extensible to additional material families by defining their constitutive models and parameter spaces, generating corresponding privileged physical states and video data, and retraining the model.
For the current model trained on these three families, an unseen material may be classified as a known family if its dynamics are similar. For example, a material exhibiting recoverable elastic deformation may be classified as Jelly, as illustrated by our expanded real-world experiments (Section 4.4 of the revised manuscript, line 478). In contrast, materials with fundamentally different dynamics (e.g., fluids) may be misclassified, leading to unreliable parameter estimates. Thus, the current model does not support open-set identification of arbitrary unseen materials.
Generalization Across Physical Conditions and Interactions.
We acknowledge that the original benchmark adopts controlled simulation settings. To broaden the evaluation, we have added OOD experiments with varying initial velocities and drop heights to assess robustness to changes in initial conditions (Appendix D.3 of the revised manuscript, line 893); introduced gripper–object interactions to extend the evaluation beyond gravity-driven deformation (Appendix D.2 of the revised manuscript, line 857); and expanded real-world experiments to additional objects (Section 4.4 of the revised manuscript, line 478).
Revisions in paper: We have added the Physical-Condition OOD Generalization experiments in Appendix D.3, lines 895–902, the Applicability to Gripper–Object Interactions experiments in Appendix D.2, lines 857–891, and the More Real-World Validation experiments in Section 4.7, lines 478–513.
We thank the reviewer for this valuable suggestion. To further assess the real-world generalization of Analytic Dynamics, we have expanded our real-world evaluation to four additional objects exhibiting diverse dynamic responses. We refer the reviewer to Section 4.7 of the revised manuscript, line 478 and watch the corresponding video comparisons via the anonymous link provided in the paper.
For each object, we used an iPhone to capture a 25-frame monocular RGB video from a frontal view, capturing the object as it is released from both hands, falls under gravity, and deforms upon contact with the tabletop. Since ground-truth material properties are unavailable, we construct a physics-based 4D generation pipeline to indirectly assess the inferred dynamics by qualitatively comparing simulated and observed deformations. Specifically, we extract masks using SAM2 and reconstruct a textured mesh from the first frame using TRELLIS-1. The pretrained Analytic Dynamics model infers the material family and parameters from the mask video. Together with the reconstructed geometry, these predictions are used to configure the MPM simulator and generate forward simulations for comparison with real observations.
As shown in Fig. 7 of the revised manuscript, line 486, our method produces simulated deformation sequences that more closely match the real observations than those produced by Optimization, Vid2Sim, and Ours (w/o alignment). These results provide further evidence of Analytic Dynamics' ability to generalize to real-world objects and its potential for practical applications.
Revisions in paper: we have added the More Real-World Validation experiments in Section 4.7, lines 478–513.