acceptodds
Under review as a conference paper at ICLR 2027

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos

Abstract

Inferring intrinsic dynamics from observed motion is essential for intelligent agents to reason about and interact with the physical world, yet remains challenging due to the fundamental gap between visual evidence and intrinsic dynamics. Existing methods either rely on costly per-scene optimization, limiting efficiency and scalability, or directly map visual evidence to intrinsic dynamics, making them prone to learning shortcuts that hinder generalization. To address this challenge, we propose Analytic Dynamics, a feed-forward framework that introduces a physics-grounded dynamics representation to bridge monocular videos and intrinsic dynamics. Specifically, we leverage privileged physical states available in simulation, including positions, displacements, and deformation gradients, which explicitly characterize object dynamics, to learn a structured dynamics representation that is difficult to discover from visual observations alone. By aligning video representations with this learned space, we equip video models with a physics-grounded inductive bias, guiding them to capture dynamics-relevant patterns for intrinsic dynamics inference. To support this research, we develop a dynamics data generation pipeline and benchmark comprising paired physical state trajectories, rendered videos, and ground-truth material labels. Extensive experiments demonstrate that Analytic Dynamics outperforms existing baselines, achieves strong generalization, and requires only 3.83 ms per inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, System Identification, Physics-Grounded Representation Learning, Intrinsic Dynamics Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NGCnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a feed-forward framework that infers physical material properties directly from monocular videos while preventing models from relying on geometric or appearance shortcuts. The approach uses a two-stage distillation process where a teacher model learns robust dynamics from simulated physical states, and a video-based student aligns its feature representations with that frozen teacher. This physics-grounded guidance enables the system to generalize effectively to unseen object shapes, maintain high inference efficiency, and transfer to real-world scenarios.

Critical strengths
  1. The problem is well-motivated problem and the approach is intuitive. The paper identifies the challenging problem of separating physical dynamics from geometry and viewpoint, and proposed physical states as a supervision.

  2. The separation between trajectory-based representation learning and video-based inference is convincing. The method transfers knowledge from a teacher that has access to simulation data to a student that operates on real-world data.

  3. I appreciate the ablation studies, e.g., the comparison between the model without alignment, the alignment-based variant, the benefit of representation distillation, etc..

  4. The method is also evaluated on unseen geometries and shorter input sequences, which supports the generalization of the method

  5. The method is efficient, e.g., 3.8ms inference time per instance.

Critical weaknesses
  1. The benchmark covers only 3 pre-defined material families, so it does not fully support the broad generalization claims and open-world problem setting. Furthermore, the simulations rely on a controlled environment with fixed settings, so it does not establish robustness to diverse physical interactions or real-world environmental conditions.

  2. Similarly, the real-world evaluation only contains 2 scenes (Bun, Burger), so it does not fully support robust real-world generalization claims.

  3. The evidence of physical grounding mainly relies on qualitative UMAP visualizations, so it does not fully serves as the evidence that the representations are physically invariant or disentangled from geometry. Overall, I think the paper might be overclaiming in multiple places.

  4. The pre-processing latency is not included. If the pre-processing is slow, the 3.8ms latency claim does not include the bottleneck of the system.

Questions for authors
  1. Does the method work beyond pre-defined material families (outside jelly, plasticine, and sand)?

  2. Does the method work in real-world beyond Bun and Burger? Perhaps yes, but I would like to see more results.

  3. Could the authors share the end-to-end latency with the breakdown to other components, e.g., pre-processing (not just 3.8ms inference time)?

3 comments
AuthorsOpenReview ↗

Response To Reviewer NGCn — Part 1

Response To Reviewer NGCn:

Thank you for your insightful comments, which are greatly valued by us. We will address your concerns below.

  • For Weakness 1 and Question 1: Open-World Material and Physical Generalization

We thank the reviewer for this valuable comment.

Generalization Beyond Pre-defined Material Families.

We select Jelly, Plasticine, and Sand as three representative material families for evaluating our method, rather than aiming to cover the full diversity of real-world materials. Importantly, our framework is extensible to additional material families by defining their constitutive models and parameter spaces, generating corresponding privileged physical states and video data, and retraining the model.

For the current model trained on these three families, an unseen material may be classified as a known family if its dynamics are similar. For example, a material exhibiting recoverable elastic deformation may be classified as Jelly, as illustrated by our expanded real-world experiments (Section 4.4 of the revised manuscript, line 478). In contrast, materials with fundamentally different dynamics (e.g., fluids) may be misclassified, leading to unreliable parameter estimates. Thus, the current model does not support open-set identification of arbitrary unseen materials.

Generalization Across Physical Conditions and Interactions.

We acknowledge that the original benchmark adopts controlled simulation settings. To broaden the evaluation, we have added OOD experiments with varying initial velocities and drop heights to assess robustness to changes in initial conditions (Appendix D.3 of the revised manuscript, line 893); introduced gripper–object interactions to extend the evaluation beyond gravity-driven deformation (Appendix D.2 of the revised manuscript, line 857); and expanded real-world experiments to additional objects (Section 4.4 of the revised manuscript, line 478).

Revisions in paper: We have added the Physical-Condition OOD Generalization experiments in Appendix D.3, lines 895–902, the Applicability to Gripper–Object Interactions experiments in Appendix D.2, lines 857–891, and the More Real-World Validation experiments in Section 4.7, lines 478–513.

  • For Weakness 2 and Question 2: More Real-World Validation

We thank the reviewer for this valuable suggestion. To further assess the real-world generalization of Analytic Dynamics, we have expanded our real-world evaluation to four additional objects exhibiting diverse dynamic responses. We refer the reviewer to Section 4.7 of the revised manuscript, line 478 and watch the corresponding video comparisons via the anonymous link provided in the paper.

For each object, we used an iPhone to capture a 25-frame monocular RGB video from a frontal view, capturing the object as it is released from both hands, falls under gravity, and deforms upon contact with the tabletop. Since ground-truth material properties are unavailable, we construct a physics-based 4D generation pipeline to indirectly assess the inferred dynamics by qualitatively comparing simulated and observed deformations. Specifically, we extract masks using SAM2 and reconstruct a textured mesh from the first frame using TRELLIS-1. The pretrained Analytic Dynamics model infers the material family and parameters from the mask video. Together with the reconstructed geometry, these predictions are used to configure the MPM simulator and generate forward simulations for comparison with real observations.

As shown in Fig. 7 of the revised manuscript, line 486, our method produces simulated deformation sequences that more closely match the real observations than those produced by Optimization, Vid2Sim, and Ours (w/o alignment). These results provide further evidence of Analytic Dynamics' ability to generalize to real-world objects and its potential for practical applications.

Revisions in paper: we have added the More Real-World Validation experiments in Section 4.7, lines 478–513.

AuthorsOpenReview ↗

Response To Reviewer NGCn — Part 2

Table 1: Quantitative comparison of physical invariance to geometric variations and sensitivity to material parameters. geometry distance measures representation differences between shapes with the same material parameters (lower is better), while physics distance measures representation differences between parameter settings for the same shape (higher is better).

MethodGeometry distance ↓Physics distance ↑
Vid2Sim0.03510.0379
Ours (w/o alignment)0.06660.0428
Ours (alignment only)0.01540.1197
Ours (Full)0.01460.1202
  • For Weakness 3: Physical Invariance and Geometry Disentanglement

To assess whether the learned representations are organized in a physically meaningful manner, we conduct a comparative analysis of learned representations by different methods. We refer the reviewer to Appendix D.1 of the revised manuscript, line 828.

Specifically, for each material, we simulate all combinations of 10 parameter levels and 10 object shapes, yielding 100 trajectories. We vary Young's modulus for Jelly and Plasticine, and friction angle for Sand, while keeping all other simulation settings unchanged. Let (z_{p,s}) denote the dynamics latent extracted from the trajectory corresponding to parameter level (p) and shape (s).

We measure two complementary properties: geometry distance, measuring representation differences between different shapes with the same material parameters (lower values indicate greater physical invariance to geometric variations), and physics distance, measuring representation differences between different material parameters under the same shapes (higher values indicate better parameter discrimination):

D_{\mathrm{geo}} &= \frac{1}{P\binom{S}{2}} \sum_{p=1}^{P}\sum_{s<t} d(z_{p,s},z_{p,t}),\\ D_{\mathrm{phys}} &= \frac{1}{S\binom{P}{2}} \sum_{s=1}^{S}\sum_{p<q} d(z_{p,s},z_{q,s}),\\ d(z,z') &= 1-\cos(z,z'). \end{aligned}$$ where (P=S=10). As shown in **Table 1 above**, **Ours (Full) consistently achieves the lowest geometry distances and highest physics distances.** This indicates that our method learns representations with greater physical invariance to geometric variations and better discrimination between different material parameter settings. **Fig. 8 of the revised manuscript, line 773**, further reveals the parameter-wise similarity structure. **For Ours (Full), representations of nearby parameter levels exhibit higher similarity, which generally decreases as the parameter gap increases.** This graded structure is less pronounced in Ours (w/o alignment) and Vid2Sim, indicating that our learned representations better preserve the continuous variations associated with material parameters. **Revisions in paper:** We have **narrowed our claims** and highlighted the corresponding changes in blue at Lines 27–30, 87–89, 104–107, and 523–525. In addition, we have added the **More Analysis of Physics-Grounded Representations** experiments in Appendix D.1, lines 828–855. * **For Weakness 4 and Question 3: End-to-End Latency and Runtime Breakdown** Thank you for raising this important point. In real-world evaluation, we measure the processing latency from the captured RGB video to the final material prediction. The pipeline consists of two stages: - **Mask Extraction:** SAM2 takes **4,625 ms** to extract object masks. - **Model Inference:** Our Student network takes **3.83 ms** for material prediction. The resulting end-to-end latency is **4,628.83 ms (4.63 s)**. Previously, we reported only network inference latency because masks are directly available in our synthetic benchmark, without requiring additional extraction. **Revisions in paper:** We have **clarified in paper that the reported 3.83 ms refers to the model forward-pass time** (lines 27-30, lines 104-107, and lines 523-525), and provided a detailed breakdown of the end-to-end latency from RGB video input to material prediction in Appendix D.8, lines 1069–1075.

I appreciate the analyses on sensitivity to material parameters, invariance to object geometry, latency, etc. Authors also provided results on four additional real-world objects, but I still have concerns about the model's ability to recognize arbitrary unseen material families. Nevertheless, the rebuttal reasonably addresses most of my concerns, so I raised my rating to weak accept.

Reviewer_sMVpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Analytic Dynamics, a feed-forward framework for inferring material families and physical parameters from monocular mask videos. A teacher learns representations from simulated particle states, which are then distilled into a video student. The authors introduce a paired simulation benchmark and report improved parameter estimation on unseen geometries, robustness to shorter observations, and preliminary real-world transfer.

Critical strengths
  1. The framework is clear and technically feasible, using simulation-only physical states during training while retaining a lightweight video model at inference.

  2. The paired dataset and geometry-disjoint evaluation provide a useful setting for studying physical parameter inference beyond familiar shapes.

  3. The alignment and physical-state ablations provide evidence that privileged supervision improves cross-shape generalization.

Critical weaknesses
  1. The methodological contribution appears primarily incremental. The approach combines standard encoders, supervised parameter prediction, and feature alignment within an established privileged-information distillation paradigm.
  2. The baseline coverage is insufficient to establish competitiveness. Generic VLMs are relatively weak references for precise numerical estimation. Including comparisons with some dedicated methods would be better.
  3. Real-world validation remains limited to two scenes without measured material parameters. PSNR cannot establish correct physical identification. More real objects and prediction under new interactions would strengthen the generalization claims.
  4. How should the method handle the fact that similar visible motion can be generated by different parameters?
Questions for authors

Please see Weaknesses.

5 comments
AuthorsOpenReview ↗

Response To Reviewer sMVp — Part 1

Table 1. Comparison with the newly added baseline Vid2Sim (partial results; see revised manuscript, line 276).

MethodJelly Acc. ↑Jelly E ↓Jelly ν ↓Plasticine Acc. ↑Plasticine E ↓Plasticine ν ↓Plasticine Yield ↓Sand Acc. ↑Sand φ ↓
Vid2Sim0.9970.2830.0381.0000.2790.0230.1680.9572.339
Ours (Full)1.0000.1750.0371.0000.1410.0210.1010.9701.777

Response To Reviewer sMVp:

Thank you for your insightful comments, which are greatly valued by us. We will address your concerns below.

  • For Weakness 1: Methodological Contribution

We thank the reviewer for the constructive comment. Our contribution lies not in proposing new network architectures, but in how we construct and transfer physically meaningful representations of intrinsic dynamics to visual learning, enabling more accurate and generalizable feed-forward inference of intrinsic dynamics from monocular videos.

Visual observations provide only limited and indirect cues about intrinsic dynamics, which often manifest only as subtle contour changes. These cues are further entangled with factors such as geometry, viewpoint, and appearance, making it difficult for the visual model to isolate dynamics-related cues, thereby limiting its generalization.

We observe that privileged physical states available in simulation, including position, velocity, and deformation gradient, provide richer, more direct, and structured dynamics information. Motivated by this insight, we propose a two-stage privileged dynamics distillation framework. Specifically, we first leverage privileged physical states to learn a physics-grounded representation. We then align the visual representation with this learned space by minimizing a feature alignment loss. Since this target space is constructed from physically informative states, aligning with it encourages the visual model to capture the corresponding dynamics-related cues from videos.

To further validate this design, we provide an additional representation analysis ( Appendix D.1 of the revised manuscript, line 828), showing that the learned representations are physically meaningful, exhibiting invariance to geometric variations (similar representations for objects with different shapes but identical dynamics), and sensitivity to material parameters (higher similarity for nearby parameter levels, which decreases as the parameter gap grows). Ablations on different privileged physical state components (Section 4.4 of the paper, line 415) further quantify their respective contributions to visual dynamics inference.

Revisions in paper: We have added the More Analysis of Physics-Grounded Representations experiments in Appendix D.1, lines 828–855.

  • For Question 2: Baseline Coverage and Competitiveness

Thank you for this constructive suggestion. Although Ours (w/o alignment) baseline already provides a closely related approach, training a feed-forward network to predict material families and their parameters directly from videos, we fully agree that comparisons with more independent, task-specific methods are important. We therefore further reproduce Vid2Sim, a dedicated method for material parameter inference. Since the original Vid2Sim assumes a fixed Neo-Hookean constitutive model and predicts only Young's modulus and Poisson's ratio, we adapt its implementation to support material-family classification and family-specific parameter regression. As shown in Tab. 1 of the revised manuscript, line 276, Ours (Full) achieves better overall performance than Vid2Sim.

We further extend the comparison with Vid2Sim to physical-condition OOD settings (Appendix D.3 of the revised manuscript, line 893), additional real-world evaluations (Section 4.7 of the revised manuscript, line 478), and a comparative analysis of the learned representations (Appendix D.1 of the revised manuscript, line 828). Our method consistently outperforms Vid2Sim across these evaluations. Taken together, these results demonstrate that privileged physical-state supervision enables us to learn physics-grounded representations, which effectively guide the inference of intrinsic material dynamics from videos.

Revisions in paper: We have added the description of the Vid2Sim baseline in Section 4.1.2, lines 310–313, and its results in Table 1, line 276. We have added the Physical-Condition OOD Generalization experiments in Appendix D.3, lines 895–902, and the More Analysis of Physics-Grounded Representations experiments in Appendix D.1, lines 828–855.

AuthorsOpenReview ↗

Response To Reviewer sMVp — Part 2

  • For Question 3: More Real-World Validation

We thank the reviewer for this valuable suggestion. To further assess the practical applicability of Analytic Dynamics, we have expanded our real-world evaluation to four additional objects exhibiting diverse dynamic responses. We refer the reviewer to Section 4.7 of the revised manuscript, line 478 and watch the corresponding video comparisons via the anonymous link provided in the paper.

For each object, we used an iPhone to capture a 25-frame monocular RGB video from a frontal view, capturing the object as it is released from both hands, falls under gravity, and deforms upon contact with the tabletop. Since ground-truth material properties are unavailable, we construct a physics-based 4D generation pipeline to indirectly assess the inferred dynamics by qualitatively comparing simulated and observed deformations. Specifically, we extract masks using SAM2 and reconstruct a textured mesh from the first frame using TRELLIS-1. The pretrained Analytic Dynamics model infers the material family and parameters from the mask video. Together with the reconstructed geometry, these predictions are used to configure the MPM simulator and generate forward simulations for comparison with real observations.

As shown in Fig. 7 of the revised manuscript, line 486, our method produces simulated deformation sequences that more closely match the real observations than those produced by Optimization, Vid2Sim, and Ours (w/o alignment). These results provide further evidence of Analytic Dynamics' ability to generalize to real-world objects and its potential for practical applications.

In addition, we further extend Analytic Dynamics to action-conditioned dynamics inference (Appendix D.2 of the revised manuscript, line 857) under gripper–object interactions, demonstrating its applicability to more diverse external stimuli beyond gravity-driven deformation.

Revisions in paper: We have added the More Real-World Validation experiments in Section 4.7, lines 478–513, and the Applicability to Gripper–Object Interactions experiments in Appendix D.2, lines 857–891.

  • For Question 4: Parameter Ambiguity under Similar Visual Motions

We agree that different material parameter combinations can produce similar visible motions. However, similar visible motions do not necessarily imply identical evolution of the underlying physical states (velocity, strain, and stress). Specifically, different material parameters define different constitutive relations between deformation and stress. Thus, even under the same deformation, they can produce different stress responses, leading to different subsequent physical-state evolution.

This motivates our two-stage privileged dynamics distillation framework. In Stage I, the Teacher leverages privileged physical states (position, velocity, and deformation gradient) to learn a representation that distinguishes different material parameter settings, even when they produce similar visible motions. In Stage II, the Student aligns monocular video representations with this learned space, transferring its physical discriminability to visual observations and encouraging the capture of subtle deformation cues associated with different material parameters.

Nevertheless, when different parameter combinations are fundamentally indistinguishable from the available observations, unique identification remains underconstrained. Future work could incorporate uncertainty-aware inference to explicitly characterize such parameter ambiguities.

Thanks to the authors for the rebuttal. It has addressed some of my concerns. Vid2Sim is somewhat outdated. Could the authors compare it with the newer ReconPhys [1]? Furthermore, I still have some concerns about the model's generalization ability to real-world scenes. The patterns of the four objects in Figure 7 are similar, indicating limited diversity. The authors could refer to https://chuanshuogushi.github.io/ReconPhys/ for more examples.

[1] ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video. ECCV 2026.

Reviewer_YqfpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Analytic Dynamics, a feed-forward framework that infers constitutive material families and material parameters from monocular videos. The framework is a two-stage dynamics distillation scheme: In Stage I, a teacher model is trained for physics-grounded representation from simulator-based physical state trajectories, including particle positions, displacements, and deformation gradients, by predicting material family and physical parameters. In Stage II, the student model is trained to align its representation with the frozen teacher while also predicting the physics intrinsics. At inference time, only the video model is retained. The authors also developed a data generation pipeline for paired physical trajectories, monocular mask videos rendered from multiple viewpoints, and ground-truth material parameters. Experiments show solid improvement on unseen geometries after representation alignment, and preliminary transfer to real-world videos.

Critical strengths
  1. Introducing a simple but meaningful way to transfer intrinsic dynamics into feed-forward video model.
  2. Interesting insight about physics-grounded distillation reduces dependence on object geometry, with effective use of figures to illustrate the point.
Critical weaknesses
  1. The OOD generalization experiment is mainly across object geometry, and the datasets are generated under a largely fixed setup. It remains unclear how much the learned representation transfer across different physical conditions.
Questions for authors
  1. The OOD evaluation is mainly across object geometry, have you evaluated the model under changes in physical conditions such as initial velocity, drop height, or contact configuration?
  2. The proposed model takes clean mask videos as input, while the VLM and Video-LLM baselines are based on RGB videos. How are the masks obtained for the real-world experiments, and how sensitive is the method to imperfect segmentation?
  3. What are the inputs and known quantities provided to the VisionLaw baseline on this benchmark?
3 comments
AuthorsOpenReview ↗

Response To Reviewer Yqfp — Part 1

Table1: OOD generalization under varying initial velocities. During training, objects are initialized with zero velocity. During testing, their initial downward velocities are uniformly sampled from [0, 0.5]. We report Log-MAE for EE and Yield, and MAE for ν and φ.

MethodJelly E ↓Jelly ν ↓Plasticine E ↓Plasticine ν ↓Plasticine Yield ↓Sand φ ↓Overall regL1 ↓
Vid2Sim0.3970.0430.3250.0240.2114.6320.100
Ours (w/o alignment)0.4240.0530.3370.0260.2033.2940.094
Ours (alignment only)0.2560.0380.1760.0220.1292.7040.068
Ours (Full)0.2640.0370.1870.0210.1372.5780.067

Table2: OOD generalization under varying drop heights. During training, objects are initialized at a fixed center height of 0.5. During testing, their center heights are uniformly sampled from [0.475, 0.525].

MethodJelly E ↓Jelly ν ↓Plasticine E ↓Plasticine ν ↓Plasticine Yield ↓Sand φ ↓Overall regL1 ↓
Vid2Sim0.5020.0410.3130.0260.3216.4020.124
Ours (w/o alignment)0.4920.0410.2780.0260.2856.0890.119
Ours (alignment only)0.3630.0420.1990.0240.2214.8550.098
Ours (Full)0.3500.0400.1890.0240.2134.8130.096

Response To Reviewer Yqfp:

We highly appreciate your acknowledgment and encouragement toward our work. Your comments are highly valuable to us.

  • For Weakness 1 and Question 1: Physical-Condition OOD Generalization

Thank you for your insightful comments. To evaluate generalization beyond geometry, we conduct two additional OOD experiments under varying initial conditions: initial velocity and drop height. During training, all objects are initialized with a fixed center height of 0.5. At test time, we independently vary the initial downward speed (uniformly sampled from [0, 0.5]) and the object-center height (uniformly sampled from [0.475, 0.525]). Both experiments follow the original test protocol, using the same three material families and 30 unseen shapes. As shown in Tables 1 and 2 above, our method achieves the lowest overall regression error under both settings, outperforming all baselines. These results demonstrate that our method generalizes more robustly to unseen initial conditions.

Beyond varying initial conditions, we further extend Analytic Dynamics to action-conditioned dynamics inference under gripper–object interactions, demonstrating its applicability to more diverse external stimuli beyond gravity-driven deformation. We refer the reviewer to Appendix D.2 of the revised manuscript, line 857 for the detailed experiments.

Revisions in paper: We have added the Physical-Condition OOD Generalization experiments in Appendix D.3, lines 895–902, and the Applicability to Gripper–Object Interactions experiments in Appendix D.2, lines 857–891.

AuthorsOpenReview ↗

Response to Reviewer Yqfp — Part 2

Table 3. Robustness to mask erosion, with erosion radii of 1, 2, and 3 pixels. We report the overall RegL1 error (↓\downarrow). Average is computed over perturbed settings, excluding Clean.

MethodClean1 px2 px3 pxAverage
Vid2Sim0.0690.0920.1260.1580.125
Ours (w/o alignment)0.0650.0820.1130.1640.120
Ours (alignment only)0.0560.0710.0960.1230.097
Ours (Full)0.0540.0710.0950.1240.097

Table 4. Robustness to mask dilation, with dilation radii of 1, 2, and 3 pixels.

MethodClean1 px2 px3 pxAverage
Vid2Sim0.0690.0780.0900.1110.093
Ours (w/o alignment)0.0650.0790.1020.1230.102
Ours (alignment only)0.0560.0640.0820.0980.081
Ours (Full)0.0540.0630.0770.0910.077
  • For Question 2: Mask Acquisition and Mask Segmentation Robustness

For real-world experiments, we obtain object masks from RGB videos using SAM2.

To evaluate sensitivity to imperfect segmentation, we conduct mask perturbations on the unseen-shape test set, including 1/2/3-pixel mask erosion and dilation to simulate boundary under- and over-segmentation, respectively. As shown in Tab. 3 and Tab. 4 above, prediction errors increase with perturbation severity, as expected. Nevertheless, Ours (Full) achieves the lowest average error across all settings. Overall, these results demonstrate that our method remains robust to imperfect perturbations.

Importantly, we have collected additional real-world examples and broadened our evaluation to provide further evidence of our method's effectiveness in practical scenarios. We refer the reviewer to Section 4.7 of the revised manuscript, line 478 and watch the corresponding video comparisons via the anonymous link provided in the paper.

Revisions in paper: We have added the Robustness to Imperfect Segmentation experiments in Appendix D.4, lines 903–910, and the More Real-World Validation experiments in Section 4.7, lines 478–513.

  • For Question 3: VisionLaw Configuration

Thank you for raising this question. We clarify the inputs and known quantities provided to the VisionLaw baseline in our benchmark below.

Visual Observations. VisionLaw uses exactly the same monocular mask videos as Analytic Dynamics for both synthetic and real-world evaluations.

3D Representation. VisionLaw employs a coupled 3DGS–Particle representation, where particles drive differentiable MPM simulation and their deformations are transferred to 3D Gaussians for differentiable rendering.

  • Synthetic data: Since only textured meshes are available, we render 30 images from randomly sampled viewpoints to reconstruct the initial 3DGS representation. We then perform volumetric particle sampling based on the corresponding mesh and establish Particle-GS bindings.
  • Real-world data: we directly use the 3D representations provided by VisionLaw.

Simulation Configuration.

  • Synthetic data: We adopt the same MPM configuration as our benchmark generation pipeline, including a grid resolution of 32332^3, a time step of 5×10−45\times10^{-4}, a substep setting of 50, and gravity of 9.8,m/s29.8,\mathrm{m/s^2}.
  • Real-world data: We follow the corresponding scene-specific configurations provided by VisionLaw.

Material Model and Parameter Initialization.

  • Synthetic data: We provide the ground-truth constitutive model and optimize only its corresponding material parameters.
  • Real-world data: We fix the constitutive model to Jelly and optimize its corresponding material parameters.

In both settings, the applicable material parameters are initialized using the same fixed values: E=105E=10^5, ν=0.3\nu=0.3, σy=5×103\sigma_y=5\times10^3, and ϕ=25∘\phi=25^\circ.

We will incorporate these details into the revised manuscript to clarify the baseline protocol and facilitate reproducibility.

Revisions in paper: We have added detailed settings for the VisionLaw baseline in Appendix C.1, lines 794–809.

Thank you for the additional experiments and clarifications. The new evaluations under varying initial velocities, drop heights, and the gripper-object interaction setting have addressed my main concern about generalization beyond object geometry. While the evaluation scope could still be broader, I do not see a fundamental issue. I therefore maintain my original rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.