acceptodds
Under review as a conference paper at ICLR 2027

GenTwin: Part-Aware Generalized Dynamics Modeling with Neural Anisotropic Springs

40 views

Abstract

Learning physically realistic object dynamics from real-world interactions is important for modeling the physical world. Existing methods typically adopt category-specific formulations for articulated and deformable dynamics, limiting their generalization to heterogeneous objects with both rigid and deformable parts. To address this, we introduce GenTwin, a part-aware framework that models generalized dynamics from real-world interaction observations within a common spring-mass representation, decomposing object dynamics into intra- and inter-part dynamics. For intra-part dynamics, we introduce Part-Adaptive Spring Configuration to adapt stiffness and topology to heterogeneous part rigidity using material priors and motion-derived rigidity. For inter-part dynamics, we connect neighboring parts with Neural Anisotropic Spring Connections, whose part-pair Neural Oriented Stiffness Fields parameterize continuous directional stiffness to capture relative motions from flexible limb swinging to constrained articulation. Experiments on 18 real-world scenarios spanning deformable, articulated, and heterogeneous objects demonstrate consistently improved geometric accuracy, reducing future prediction Chamfer distance by approximately 28.6%, 27.3%, and 50.0%, respectively, over the best competing category-specific methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamics Modeling, Spring-Mass Model, Deformable Objects, Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DvFtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a part-aware framework based on a spring-mass system for learning physically realistic object dynamics from real-world observations. The method builds upon an existing differentiable physics simulator and focuses on estimating physical parameters through a combination of intra-part and inter-part dynamics modeling. Specifically, the intra-part module adapts spring configurations to capture varying degrees of deformation and rigidity, while the inter-part module incorporates directional stiffness to model constrained relative motion between object components. Experimental results demonstrate that the proposed method outperforms existing baselines across deformable, articulated, and heterogeneous object scenarios.

Critical strengths
  • Strong experimental performance. The proposed method consistently outperforms existing baselines across deformable, articulated, and heterogeneous object scenarios, demonstrating improved geometric accuracy and future dynamics prediction.
  • Effective modeling of inter-part dynamics. The proposed inter-part spring connections effectively capture constrained relative motion between object components, improving the modeling of articulated structures while accommodating flexible inter-part interactions.
Critical weaknesses

This paper primarily addresses physics parameter estimation by extending the spring-mass framework introduced in PhysTwin [1]. The proposed method incorporates part-aware spring configurations and neural anisotropic stiffness to model deformable, articulated, and heterogeneous objects within a unified representation. Although the experimental results demonstrate promising improvements, my main concerns are related to the insufficient description of the physical simulation, limited baseline comparisons, and the novelty of the proposed formulation.

Weaknesses and Questions

1. Unclear initialization of spring rest lengths.

How is the rest length ℓe\ell_e in Equation (7) determined? Is it directly calculated from the initial observed geometry, estimated from the interaction observations, or manually specified? In particular, how are the rest lengths of the newly introduced long-range springs initialized?

2. Insufficient details about the physical dynamics formulation.

In Section 3.4, the authors briefly mention that gravity, contact, and friction forces contribute to the particle dynamics. However, Equation (7) only describes the strain-based spring force. What are the mathematical formulations and numerical treatments of the contact and friction forces? Furthermore, the paper does not explicitly provide the complete equations of motion governing the evolution of particle positions and velocities. A more detailed description of the physical dynamics and numerical integration scheme would improve the technical clarity and reproducibility of the method.

3. Missing comparison with SoMA [2].

The paper cites SoMA [2] but does not provide an experimental comparison. SoMA also aims to learn physically realistic object dynamics from real-world interactions, although it adopts a neural dynamics formulation rather than an explicit spring-mass model. A comparison on applicable deformable-object scenarios would help demonstrate the relative advantages of the proposed physics-based framework, particularly in terms of prediction accuracy and generalization to unseen interactions. While I acknowledge that SoMA does not explicitly model inter-part constraints in the same manner as the proposed method, it remains a relevant baseline for evaluating generalized object dynamics modeling.

4. Insufficient details about the PhysTwin baseline on the newly collected scenarios.

The paper introduces 11 self-collected articulated and heterogeneous interaction scenarios. However, it is unclear how PhysTwin [1] is adapted and optimized for these new scenarios. Does PhysTwin use exactly the same training observations, reconstructed geometry, tracking data, and controller trajectories as the proposed method? Are its topology and physical parameters re-optimized for each scenario using comparable optimization budgets?

5. Clarification of controller trajectories and evaluation fairness. Are the controller trajectories used during future prediction obtained from the held-out ground-truth observations, or are they estimated independently? Furthermore, do all baseline methods use the same controller points and trajectories during evaluation? If not, how are differences in control inputs accounted for to ensure a fair comparison?

6. Limited novelty and insufficient ablation studies to isolate the contributions over PhysTwin.

In the Introduction, the authors raise the question: "Can we achieve generalized dynamics modeling across deformable, articulated, and heterogeneous objects within a common physical representation?" However, PhysTwin [1] has already introduced a unified spring-mass representation with optimized spring connectivity and stiffness. Although PhysTwin primarily focuses on deformable objects, its underlying representation is not inherently restricted to such objects.

The proposed method appears to extend PhysTwin primarily through part-aware spring configuration, rigidity-adaptive long-range connections, and direction-dependent inter-part stiffness. While these modifications are reasonable and potentially effective, the paper does not sufficiently establish their individual and combined contributions relative to the original PhysTwin formulation.

In particular, Table 2 separately evaluates the variants "w/ dense local" and "w/o anisotropy", but does not report a variant combining both modifications. I suggest adding an ablation that simultaneously replaces the adaptive long-range springs with dense local connections and removes the neural anisotropic stiffness. This configuration would be closer to the conventional spring-mass formulation in PhysTwin, although differences in material priors, part-aware parameterization, and preprocessing may still remain.

Such an ablation would help clarify how much of the improvement comes specifically from the proposed spring topology and directional stiffness modeling. Ideally, the comparison should use the same preprocessing pipeline and optimization settings to isolate the effects of the proposed dynamics representation.

References

[1] Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7219–7230, 2025.
https://openaccess.thecvf.com/content/ICCV2025/html/Jiang_PhysTwin_Physics-Informed_Reconstruction_and_Simulation_of_Deformable_Objects_from_Videos_ICCV_2025_paper.html

[2] Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, and Jiangmiao Pang. SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-Body Manipulation. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:47834–47850, 2026.
https://proceedings.mlr.press/v306/huang26bh.html

Questions for authors

Please refer to the weaknesses.

8 comments
AuthorsOpenReview ↗

Response to reviewer DvFt (Part1)

Thank you for your careful review and constructive feedback. We have addressed your comments below and updated the revised manuscript, with revisions highlighted in blue.

[WQ1] Unclear initialization of spring rest lengths.

How is the rest length in Equation (7) determined? Is it directly calculated from the initial observed geometry, estimated from the interaction observations, or manually specified?

Thank you for requesting this clarification. We have clarified the initialization of spring rest lengths in Sec. 3.4. For local springs, CMA-ES searches the connection radius and maximum neighbor count, which determine the connected particle pairs. For each connected pair (i,j)(i,j), the rest length is computed directly as the Euclidean distance between the two sampled points in the initial object geometry P0\mathcal{P}^0:

ℓ(i,j)=∣xj0−xi0∣2.\ell_{(i,j)}=\\|\mathbf{x}_j^0-\mathbf{x}_i^0\\|_2.

In particular, how are the rest lengths of the newly introduced long-range springs initialized?

For long-range springs, we connect sampled points in the initial object geometry within the same near-rigid part pp, using their Euclidean distances as the rest lengths. Following Eq. (4), only connections within the prescribed distance range, scaled by the part extent Lp=max⁡i,j∈Vp∣xi0−xj0∣2L_p=\max_{i,j\in\mathcal{V}_p}\\|\mathbf{x}_i^0-\mathbf{x}_j^0\\|_2, are retained:

β−Lp≤∣xi0−xj0∣2≤β+Lp,i,j∈Vp.\beta^-L_p\leq\\|\mathbf{x}_i^0-\mathbf{x}_j^0\\|_2\leq\beta^+L_p,\qquad i,j\in\mathcal{V}_p.

In our default implementation, the prescribed normalized range is (β−,β+)=(0.4,0.405)(\beta^-,\beta^+)=(0.4,0.405). To evaluate this parameter choice, we added a long-range spring distance ablation on the 11 self-captured scenarios in Sec. 4.3. The table below summarizes future prediction results for different values of β−\beta^-.

β−\beta^-CD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
0.20.0110.0170.75126.7810.9650.036
0.60.0100.0150.77227.1270.9660.034
0.4 (Ours)0.0090.0150.77727.3280.9670.033

As shown in the table, springs near 0.4Lp0.4L_p strike a favorable balance across all metrics. Shorter springs provide weaker rigidity constraints, while longer springs yield comparable performance.

AuthorsOpenReview ↗

Response to reviewer DvFt (Part2)

[WQ2] Insufficient details about the physical dynamics formulation.

Thank you for requesting these details. We have expanded Appendix B.5 with the complete equations and numerical treatment. We summarize the main updates below.

Particle Dynamics. We advance the spring-mass system using substeps indexed by nn, each with duration Δτ\Delta\tau. The spring and axial damping forces in Eq. (7), together with gravity, update particle velocities as

Fin=∑j∈N(i)fi←jn+mig,v^i=exp⁡(−βΔτ)(vin+ΔτmiFin),\mathbf{F}_i^n=\sum_{j\in\mathcal{N}(i)}\mathbf{f}_{i\leftarrow j}^n+m_i\mathbf{g},\qquad\widehat{\mathbf{v}}_i=\exp(-\beta\Delta\tau)\left(\mathbf{v}_i^n+\frac{\Delta\tau}{m_i}\mathbf{F}_i^n\right),

where N(i)\mathcal{N}(i) denotes spring-connected neighbors, g\mathbf{g} is gravitational acceleration, and β\beta controls drag damping.

What are the mathematical formulations and numerical treatments of the contact and friction forces?

Contact and Friction. Contact is activated for collision-enabled pairs with distinct collision labels when their separation falls below the collision threshold. For a contacting pair (i,j)(i,j), let nij\mathbf{n}_{ij} be the unit normal from jj to ii. We decompose relative velocity as

u=v^i−v^j,un=(u⊤nij)nij,ut=u−un.\mathbf{u}=\widehat{\mathbf{v}}_i-\widehat{\mathbf{v}}_j,\qquad\mathbf{u}_n=(\mathbf{u}^{\top}\mathbf{n}_{ij})\mathbf{n}_{ij},\qquad\mathbf{u}_t=\mathbf{u}-\mathbf{u}_n.

The impulse applied to particle ii is

aij=max⁡(0,,1−μ(1+e)∣un∣2max⁡(∣ut∣2,ε)),Jij=−(1+e)un+(1−aij)utmi−1+mj−1.a_{ij}=\max\left(0,\\,1-\frac{\mu(1+e)\\|\mathbf{u}_n\\|_2}{\max(\\|\mathbf{u}_t\\|_2,\varepsilon)}\right),\qquad\mathbf{J}_{ij}=-\frac{(1+e)\mathbf{u}_n+(1-a_{ij})\mathbf{u}_t}{m_i^{-1}+m_j^{-1}}.

Here, ee and μ\mu are restitution and friction coefficients, and ε\varepsilon prevents division by zero. The normal component produces restitution, while the tangential component opposes sliding under Coulomb friction. We average impulses over the active contact set Ci\mathcal{C}_i:

v‾i=v^i+1mi∣Ci∣∑j∈CiJij.\overline{\mathbf{v}}_i=\widehat{\mathbf{v}}_i+\frac{1}{m_i|\mathcal{C}_i|}\sum_{j\in\mathcal{C}_i}\mathbf{J}_{ij}.

Without active contacts, v‾i=v^i\overline{\mathbf{v}}_i=\widehat{\mathbf{v}}_i.

Furthermore, the paper does not explicitly provide the complete equations of motion governing the evolution of particle positions and velocities.

Ground Contact and State Update. Ground collisions use normal restitution and Coulomb friction, with position updates split at the estimated time of impact. Without ground contact, the state update is

vin+1=v‾i,xin+1=xin+Δτvin+1.\mathbf{v}_i^{n+1}=\overline{\mathbf{v}}_i,\qquad\mathbf{x}_i^{n+1}=\mathbf{x}_i^n+\Delta\tau\mathbf{v}_i^{n+1}.

The complete ground-contact and position-update equations are included in Appendix B.5.

AuthorsOpenReview ↗

Response to reviewer DvFt (Part3)

[WQ3] Missing comparison with SoMA. A comparison on applicable deformable-object scenarios would help demonstrate the relative advantages of the proposed physics-based framework, particularly in terms of prediction accuracy and generalization to unseen interactions.

Thank you for your suggestion. We have adapted SoMA's official code release to the seven deformable-object scenarios from the PhysTwin dataset. Implementation details and qualitative and quantitative comparisons are provided in Appendix C.4. The key settings are summarized below.

For preprocessing, we convert the same RGB-D observations, camera calibrations, and object tracks used by GenTwin to SoMA's scene format, retaining the same train/test splits. For controller setting, we use the shared 3D hand trajectories to drive autoregressive prediction without reinitialization at the train/test boundary. For optimization, we train a separate model per scenario on four NVIDIA A100 GPUs, following SoMA's two-stage procedure with 15 epochs of coarse dynamics training followed by 10 epochs of frame-to-frame training, and perform inference on a single A100 GPU. For evaluation, we compute geometric and tracking metrics on the predicted object Gaussian centers throughout the rollout, assigning each reference track to its nearest Gaussian in the initial frame and keeping this correspondence fixed.

The table below reports results averaged over the seven deformable-object scenarios.

EvaluationMethodCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
Reconstruction & ResimulationSoMA0.0140.0290.73226.8420.9480.061
Reconstruction & ResimulationOurs0.0040.0070.86527.6870.9510.023
Future PredictionSoMA0.0390.0920.41821.3010.9100.121
Future PredictionOurs0.0100.0200.74525.0970.9410.048

Under this single-sequence training setting, SoMA achieves worse performance than GenTwin. Through discussions with SoMA's first author, we learned that its original configuration was designed for joint learning from multiple interaction sequences, and training on a single sequence may require substantial manual hyperparameter tuning. These results suggest that GenTwin performs better than SoMA under a single-interaction training configuration.

Reviewer_bT3ZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents GenTwin, a part-aware framework that models generalized object dynamics (articulated, deformable, and heterogeneous). (1) For intra-part dynamics, Part-Adaptive Spring Configuration adapts stiffness using VLM priors, and adapts topology using part rigidity. (2) For inter-part dynamics, Neural Anisotropic Spring Connections connects springs across parts and Neural Oriented Stiffness Field predicts stiffness residuals from the spring directions. Experiments demonstrate that GenTwin outperforms prior methods across deformable, articulated, and heterogeneous object categories.

Critical strengths

While previous methods only model articulated dynamics or deformable dynamics, the proposed GenTwin models generalized dynamics including articulated, deformable, and heterogeneous objects. The proposed components, Part-Adaptive Spring Configuration and Neural Anisotropic Spring Connections, enables the generalized dynamics. The quantitative and qualitative evaluation on diverse object categories validate that GenTwin models generalized dynamics well, especially on future prediction in Tab. 1 and unseen interaction in qualitative results.

Critical weaknesses
  • Tab. 2 provides ablation study to validate the proposed components. However, the performance degradation of ablation variants are small compared to the performance gap in Tab. 1. Compared to the PhysTwin, which component or components provide performance gain?
  • To my knowledge, the zero-order optimization is sensitive to random initialization. How robust is GenTwin? The standard deviation over random seeds might be helpful.
Questions for authors

How about computational cost of GenTwin during optimization and inference?

5 comments
AuthorsOpenReview ↗

Response to reviewer bT3Z (Part1)

Thank you for your careful review and constructive feedback. We have addressed your comments below and updated the revised manuscript, with revisions highlighted in blue.

[W1.1] The performance degradation of ablation variants are small compared to the performance gap in Tab. 1.

Thank you for pointing this out. The original Table 1 reports results by object category, whereas the original Table 2 averages over 11 self-captured scenarios, so their performance gaps are not directly comparable. We have therefore added PhysTwin results on the same 11 scenarios to the revised Table 2 and reorganized the ablations into Variants 1–5. Their configurations are summarized below. Variants 3, 4, and 5 correspond to the previous "w/ dense local," "w/o material," and "w/o anisotropy" settings, respectively.

VariantMaterial PriorsDense Local SpringsLong-Range SpringsAnisotropy
Variant 1✓
Variant 2✓✓
Variant 3✓✓✓
Variant 4✓✓
Variant 5✓✓
Full model✓✓✓

Among the newly added variants, Variant 1 augments PhysTwin with only part-level spring configurations and material priors. Variant 2 additionally uses dense local springs, yielding a configuration closer to the conventional spring-mass formulation in PhysTwin. The following table reports future prediction results for PhysTwin, all five variants, and the full model, averaged over the same 11 self-captured scenarios.

VariantCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
PhysTwin0.0150.0270.70025.3860.9610.042
Variant 10.0120.0220.73626.2510.9630.039
Variant 20.0110.0220.73426.2900.9630.039
Variant 30.0100.0190.75026.7780.9650.036
Variant 40.0100.0170.76127.0970.9660.035
Variant 50.0110.0200.73726.2250.9630.039
Full model0.0090.0150.77727.3280.9670.033

[W1.2] Compared to the PhysTwin, which component or components provide performance gain?

We examine the contributions through three groups of comparisons:

  1. Part-level spring configurations and material priors. Adding these alone (Variant 1) reduces future prediction CD from 0.015 to 0.012 and Track error from 0.027 to 0.022 compared with PhysTwin. This supports the value of part-aware modeling for articulated objects with substantial inter-part motion and heterogeneous objects with contrasting part materials.

  2. Neural Anisotropic Spring Connections. Adding anisotropy reduces future prediction CD and Track errors by approximately 9.1% and 13.6% with dense local springs (Variant 3 vs. Variant 2), and by 18.2% and 25.0% with long-range springs (full model vs. Variant 5). Without anisotropy, Variants 1, 2, and 5 achieve similar performance across different topologies, reinforcing the key role of directional stiffness modeling.

  3. Part-Adaptive Spring Configuration. With anisotropic inter-part connections in both variants, the full model's rigidity-adaptive long-range springs reduce future prediction CD and Track errors by approximately 10.0% and 21.1% compared with the dense local springs in Variant 3. Fig. 9(d) further shows better preservation of the feather-duster handle's rigidity.

In summary, the performance gains over PhysTwin mainly come from three aspects: part-level spring configurations and material priors first provide a part-aware modeling foundation for challenging scenarios; Neural Anisotropic Spring Connections then play a key role in capturing large relative motions between parts, bringing substantial performance improvements; and Part-Adaptive Spring Configuration further improves the modeling of rigid-part dynamics.

AuthorsOpenReview ↗

Response to reviewer bT3Z (Part2)

[W2] To my knowledge, the zero-order optimization is sensitive to random initialization. How robust is GenTwin? The standard deviation over random seeds might be helpful.

Thank you for suggesting a quantitative analysis of seed sensitivity. We conducted five runs per scenario on three representative cases: double_lift_sloth (deformable), double_close_drawer_11 (articulated), and single_push_duster_03 (heterogeneous). We vary only the CMA-ES random seed, keeping the input data, stiffness bounds, search settings, and differentiable refinement initialization fixed within each scenario. Each run uses 20 CMA-ES generations followed by 200 differentiable updates. The table below reports future prediction metrics as mean ± sample standard deviation over the five runs per scenario.

ScenarioCD (m) ↓Track (m) ↓IoU↑PSNR↑SSIM↑LPIPS↓
double_lift_sloth0.0130 ± 0.00050.0262 ± 0.00080.794 ± 0.01425.086 ± 0.4190.9231 ± 0.00220.0601 ± 0.0031
double_close_drawer_110.0072 ± 0.00020.0105 ± 0.00050.939 ± 0.00229.971 ± 0.1530.9713 ± 0.00060.0270 ± 0.0007
single_push_duster_030.0067 ± 0.00020.0152 ± 0.00030.820 ± 0.01032.930 ± 0.1130.9743 ± 0.00040.0205 ± 0.0003

Fig. 19 shows the corresponding best-so-far CMA-ES losses, with the final generations enlarged. Across the reported runs, the small variations in future prediction metrics indicate consistent performance under the evaluated CMA-ES seed changes.

[Q1] How about computational cost of GenTwin during optimization and inference?

Thank you for asking about computational cost. We profiled GenTwin on the same three representative scenarios as [W2], each using one NVIDIA L40 GPU. Optimization consists of 20 CMA-ES generations followed by 200 Adam updates. The measured costs are summarized below.

Each stage entry reports time / peak CPU memory (RSS) / sampled peak GPU memory.

ScenarioCMA-ESRefinementInferenceSimulation FPS
double_lift_sloth16.79 min / 1.759 GiB / 2.434 GiB83.95 min / 2.923 GiB / 1.309 GiB38.50 s / 1.983 GiB / 1.205 GiB6.41
double_close_drawer_1132.37 min / 1.792 GiB / 2.495 GiB81.15 min / 2.513 GiB / 1.231 GiB43.61 s / 1.963 GiB / 1.146 GiB5.53
single_push_duster_038.33 min / 1.710 GiB / 2.094 GiB86.45 min / 2.504 GiB / 1.020 GiB42.77 s / 1.950 GiB / 0.963 GiB5.75

Optimization and inference times cover the complete programs, including data loading and output. Simulation FPS is measured over the forward loop, including stiffness-field evaluation and trajectory transfer to CPU. Preprocessing and Gaussian rendering are excluded.

AuthorsOpenReview ↗

Gentle follow-up regarding our rebuttal

Dear Reviewer bT3Z,

Following up on our rebuttal submitted a couple of days ago, we wanted to kindly check if our detailed response has satisfactorily resolved the concerns you raised.

Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you might have.

Best regards, The Authors

Reviewer_zTcKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GenTwin, a part-aware framework for modeling deformable, articulated, and heterogeneous object dynamics from multi-view RGB-D interaction videos using a unified spring-mass representation. Part-Adaptive Spring Configuration combines material priors and motion-derived rigidity to configure intra-part stiffness and topology, while Neural Anisotropic Spring Connections model inter-part motion through directional stiffness. Parameters are optimized using CMA-ES followed by differentiable simulation. Experiments cover 18 real-world scenarios.

Critical strengths

The paper addresses a meaningful problem through a unified representation of diverse object dynamics. The decomposition into intra-part and inter-part dynamics is intuitive, and the proposed components form a coherent framework. Category-averaged results show improved geometric accuracy over the compared baselines. Ablations support the benefits of material priors, rigidity-adaptive topology, and directional stiffness.

Critical weaknesses

W1. Insufficient isolation of the benefit of the neural tensor parameterization. The Neural Oriented Stiffness Field predicts a direction-dependent symmetric tensor and projects it onto the same direction to obtain scalar stiffness. The isotropic ablation supports directional dependence, but does not isolate the benefit of this tensor output over direct scalar prediction. Comparisons with a constant anisotropic tensor per part pair and a scalar-output field using matched directional encoding, comparable capacity, and optimization settings would better establish the contribution of the proposed parameterization.

W2. Confounded comparisons with articulated reconstruction baselines. Appendix C.2 acknowledges that ArtGS and GaussianArt use two endpoint states and temporal interpolation/extrapolation, whereas GenTwin uses interaction sequences and controller-conditioned simulation. These differences limit attribution of the performance gap to the dynamics representation, particularly under nonuniform manipulation speeds. A controlled comparison or separate evaluation of articulation reconstruction and trajectory prediction would be more informative. This concern specifically affects the ArtGS/GaussianArt comparisons; Table 1 also reports improvements over PhysTwin, although shared preprocessing inputs should be confirmed given the changes in Appendix A.3.

W3. Limited evidence on robustness to upstream errors. Geometry, tracking, and part segmentation affect rigidity estimation and spring construction. Tracking residuals determine topology through a hard rigidity threshold, while incorrect material predictions may restrict stiffness optimization to unsuitable ranges. The tracking filters, material-prior ablations, and alternative-prior comparisons are useful, but do not establish sensitivity to controlled input errors. Quantitative evaluations would clarify the framework's operating range, especially when segmentation mistakes mix independently moving parts or tracking noise alters rigidity classification.

Questions for authors

Q1. How does the Neural Oriented Stiffness Field compare with a learnable constant anisotropic stiffness tensor for each part pair in the same anchor-local frame?

Q2. Would a scalar-output field using the same directional encoder, comparable capacity, and stiffness decoding achieve similar performance to the tensor-based parameterization?

Q3. Can the authors provide an articulated-object comparison that better controls observation availability and motion conditioning, and confirm whether PhysTwin uses the same initial geometry and tracking inputs?

Q4. How do controlled perturbations to reconstructed geometry and 3D tracks affect rigidity classification, topology selection, and final simulation accuracy?

Q5. How does performance change under part merging/splitting or incorrect material labels and stiffness bounds, beyond the existing material-prior removal and replacement experiments?

5 comments
AuthorsOpenReview ↗

Response to reviewer zTcK (Part1)

Thank you for your careful review and constructive feedback. We have addressed your comments below and updated the revised manuscript, with revisions highlighted in blue.

[W1] Insufficient isolation of the benefit of the neural tensor parameterization.

Thank you for your suggestion. We added two ablation variants, Constant tensor and Neural scalar, and compared both with Neural tensor (Ours) on 11 self-captured scenarios. The results are analyzed in Sec. 4.3 of the revised manuscript, with complete metrics in Appendix D.4. We detail their implementations and discuss the results in our responses to Q1 and Q2 below.

[Q1] How does the Neural Oriented Stiffness Field compare with a learnable constant anisotropic stiffness tensor for each part pair in the same anchor-local frame?

Constant tensor: We replace the neural field with six learnable coefficients defining a constant symmetric tensor ΔKpq\Delta\mathbf{K}_{pq} for each part pair (p,q)(p,q) in the same anchor-local frame. Using the same direction vector, we compute the stiffness residual as Δket=(d~et)⊤ΔKpqd~et\Delta k_e^t=(\widetilde{\mathbf{d}}_e^t)^\top\Delta\mathbf{K}_{pq}\widetilde{\mathbf{d}}_e^t. Future prediction results on the 11 scenarios are summarized below.

VariantsCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
Constant tensor0.0110.0210.73926.2540.9630.039
Neural tensor (Ours)0.0090.0150.77727.3280.9670.033

The constant tensor performs worse across all six metrics, suggesting that a single shared quadratic form is too restrictive to capture the effective anisotropy of inter-part spring groups. Our neural tensor instead predicts coefficients conditioned on each spring’s anchor-local direction. This allows differently oriented springs to use distinct tensor coefficients, overcoming the restriction of a shared quadratic form.

[Q2] Would a scalar-output field using the same directional encoder, comparable capacity, and stiffness decoding achieve similar performance to the tensor-based parameterization?

Neural scalar: We use the same directional encoder, MLP depth, hidden width, and activation functions, replacing only the six-coefficient tensor output and quadratic projection with a scalar output that directly predicts the stiffness residual, Δket=ϕpq(d~et)\Delta k_e^t=\phi_{pq}(\widetilde{\mathbf{d}}_e^t). All other settings, including initialization, stiffness decoding, and optimization, remain unchanged. Future prediction results on the 11 scenarios are summarized below.

VariantsCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
Neural scalar0.0100.0180.76126.9080.9650.036
Neural tensor (Ours)0.0090.0150.77727.3280.9670.033

Our neural tensor improves all six metrics over direct scalar prediction. This suggests that the tensor parameterization provides a useful inductive bias for learning anisotropic stiffness distributions, compared with directly learning the mapping from direction to stiffness residual.

AuthorsOpenReview ↗

Response to reviewer zTcK (Part2)

[W2] Confounded comparisons with articulated reconstruction baselines.

Thank you for highlighting this confound. To better control observation availability and motion conditioning, we have adapted ArtGS and GaussianArt through multi-frame joint training on all training frames and controller-bound inference using prescribed hand trajectories, including those extracted from held-out observations for future prediction. The revised results are reported in Table 1 of our revised manuscript.

[Q3.1] Can the authors provide an articulated-object comparison that better controls observation availability and motion conditioning?

Multi-frame joint training. Both methods use three calibrated RGB-D views from every training frame and follow their respective warm-up procedures to initialize the Gaussian representation and articulation. For each of the TtrT_{\mathrm{tr}} training frames, we introduce a learnable articulation coordinate qtq_t, shared across camera views, and jointly optimize these coordinates with the Gaussian and articulation parameters Φ\Phi under the training observations Dtr\mathcal{D}_{\mathrm{tr}}:

(Φ∗,qt∗)=arg,min⁡Φ,qtLtrain(Φ,qt;Dtr),s.t.q0=0,qTtr−1=1.\left(\Phi^*,\\{q_t^*\\}\right)=\operatorname*{arg\\,min}_{\Phi,\\{q_t\\}}\mathcal{L}_{\mathrm{train}}\left(\Phi,\\{q_t\\};\mathcal{D}_{\mathrm{tr}}\right),\qquad\text{s.t.}\quad q_0=0,\quad q_{T_{\mathrm{tr}}-1}=1.

We initialize qt=t/(Ttr−1)q_t=t/(T_{\mathrm{tr}}-1) and optimize the intermediate coordinates to fit the observations, allowing them to capture changes in motion speed and pauses.

Controller-bound inference. After training, we fix the Gaussians and articulation parameters and select the hand point closest to the visible moving part in the first frame, denoting its trajectory by ht\mathbf{h}_t. We sample KK distinct Gaussians from that part and keep the hand-point identity and Gaussian indices fixed. Along the articulation path,

xi(s)=Ri(s)xi(0)+ti(s),di0=∣xi(0)−h0∣2.\mathbf{x}_i(s)=\mathbf{R}_i(s)\mathbf{x}_i(0)+\mathbf{t}_i(s),\qquad d_i^0=\\|\mathbf{x}_i(0)-\mathbf{h}_0\\|_2.

The hand–Gaussian binding term for articulated inference is

min⁡s^t∑t=0T−11K∑i=1K(∣xi(s^t)−ht∣2−di0)2,s.t.s^0=0.\min_{\\{\widehat{s}_t\\}}\sum_{t=0}^{T-1}\frac{1}{K}\sum_{i=1}^{K}\left(\\|\mathbf{x}_i(\widehat{s}_t)-\mathbf{h}_t\\|_2-d_i^0\right)^2,\qquad\text{s.t.}\quad\widehat{s}_0=0.

Here, TT is the full sequence length. This term estimates inference-time articulation states s^t\widehat{s}_t by encouraging each sampled Gaussian to preserve its initial distance to the moving hand, guiding articulated predictions according to the prescribed controller trajectory.

Articulated results. The table below summarizes the revised results from Table 1, averaged over five articulated scenarios.

EvaluationMethodCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓
Reconstruction & ResimulationArtGS0.0120.0090.87128.9480.9630.047
GaussianArt0.0140.0110.81127.8500.9580.053
Ours0.0070.0090.91727.7690.9520.038
Future PredictionArtGS0.0130.0130.84827.0640.9570.052
GaussianArt0.0150.0150.79225.5400.9520.059
Ours0.0080.0120.90626.5240.9520.044

Under this controlled setting, GenTwin achieves lower CD and equal or lower tracking error than both articulated baselines, while remaining competitive in rendering quality. It trails articulation-specific methods only in PSNR and SSIM, possibly because their access to all training frames enables better optimization of Gaussian appearance. Fig. 5 further illustrates slightly better tracking alignment during lid rotation.

[Q3.2] confirm whether PhysTwin uses the same initial geometry and tracking inputs?

Yes. For the seven deformable-object scenarios, both PhysTwin and GenTwin use the official PhysTwin dataset as provided, without applying our preprocessing pipeline (Appendix A.3). For the 11 self-captured scenarios, both methods share the same preprocessed partial observations, initial reconstructed geometry, 3D object tracks, hand trajectories, and train/test splits.

AuthorsOpenReview ↗

Response to reviewer zTcK (Part3)

[W3] Limited evidence on robustness to upstream errors.

Thank you for this valuable suggestion. We selected one representative scenario from each object category: double_lift_sloth, double_close_drawer_11, and single_push_duster_03. We introduced controlled perturbations of varying severity to geometric priors, tracking trajectories, part segmentation, and material recognition. The settings and results are detailed in our responses to Q4 and Q5 below, with further details in Sec. 4.4 and Appendix D.5 of the revised manuscript.

[Q4] How do controlled perturbations to reconstructed geometry and 3D tracks affect rigidity classification, topology selection, and final simulation accuracy?

Geometric Prior Perturbation. We perturb the generative prior meshes with local stretching of 2.5%, 5%, and 10% of each part's extent and global rotations of 2.5∘2.5^\circ, 5∘5^\circ, and 10∘10^\circ at Levels 1–3. We retrain the first-frame static Gaussian representation after each perturbation. Since rigidity classification is based on part-level tracks, the table below reports changes in effective spring counts alongside future prediction metrics averaged over the three scenarios at each perturbation level.

VariantsCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓Spring Count
Ours0.0080.0170.85729.4640.9580.035105947
Level 10.0080.0170.84829.4160.9570.035108288
Level 20.0090.0170.82729.0650.9550.039103797
Level 30.0100.0190.80828.8560.9540.04386050

At Level 3, future prediction CD increases by 0.002 and PSNR decreases by 0.608 dB. Prior misalignment can degrade the initial Gaussian reconstruction and leave displaced sample points with fewer valid spring connections. Despite these effects, the limited performance degradation indicates robustness to the geometric prior perturbations.

Tracking Trajectory Perturbation. We perturb the tracking trajectories with Gaussian noise of 1, 2.5, and 5 mm at Levels 1–3, respectively. The table below reports the part-level residual deformation variance epe_p for two representative scenarios, rounded to six decimal places.

Scenariopart_idLevel 1Level 2Level 3
double_close_drawer_1100.0003560.0007520.002172
double_close_drawer_1110.0044920.0048340.006088
single_push_duster_0300.0010090.0013480.002613
single_push_duster_0310.0209940.0213440.022644

Although epe_p increases with tracking noise, none of these parts crosses the rigidity threshold ϵrigid=0.01\epsilon_{\mathrm{rigid}}=0.01. Their rigidity classifications and corresponding selection of long-range spring augmentation therefore remain unchanged across all three levels. The following table reports effective spring counts and future prediction metrics averaged over the three scenarios at each perturbation level.

VariantsCD↓Track↓IoU↑PSNR↑SSIM↑LPIPS↓Spring Count
Ours0.0080.0170.85729.4640.9580.035105947
Level 10.0080.0160.85629.3250.9570.036101750
Level 20.0090.0170.85228.8670.9550.03794622
Level 30.0100.0170.84429.0730.9550.039106904

At Level 3, CD increases by 0.002 and PSNR decreases by 0.391 dB, indicating limited degradation under tracking noise, which can be attributed to the robustness of differentiable physics simulation to such noise. Meanwhile, the effective spring count remains largely unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary

We sincerely thank all reviewers for their constructive feedback. We have revised the manuscript, with changes highlighted in blue.

The reviewers' main concerns involve additional ablations to clarify contributions (zTcK, bT3Z, DvFt), the fairness and completeness of baseline comparisons (zTcK, DvFt), robustness to upstream errors and random seeds (zTcK, bT3Z), and implementation details and computational cost (bT3Z, DvFt).

1. Additional ablations to clarify contributions. We compared five ablation variants, including the joint ablation requested by Reviewer DvFt. Results on 11 self-captured scenarios (Sec. 4.3) show that part-level configurations and material priors provide a modeling foundation, neural anisotropic connections play a key role in capturing inter-part motion, and rigidity-adaptive long-range springs further preserve part rigidity. Compared with the requested joint ablation, the full model reduces future prediction CD and Track errors by 18.2% and 31.8%. Details are provided in our responses to Reviewers bT3Z (W1.1-W1.2) and DvFt (WQ6).

We also ablated the neural tensor parameterization and long-range spring distance scale (Tables 4 and 5). Our neural tensor outperforms both a constant tensor per part pair and direct neural scalar prediction: direction-dependent coefficients increase flexibility beyond a shared quadratic form, while the gains over scalar prediction suggest a useful inductive bias from the quadratic directional structure. Long-range springs near 0.4Lp0.4L_p strike a favorable balance across metrics compared with 0.2Lp0.2L_p and 0.6Lp0.6L_p. Details are provided in our responses to Reviewers zTcK (W1, Q1-Q2) and DvFt (WQ1).

2. Fairness and completeness of baseline comparisons. As requested by Reviewer zTcK, we adapted ArtGS and GaussianArt through multi-frame joint training on all training frames and controller-bound inference using prescribed hand trajectories, including those extracted from held-out ground-truth observations for future prediction. GenTwin retains competitive rendering quality while achieving lower geometric and equal or lower tracking errors (Table 1; Figs. 5 and 20). Details are provided in our response to Reviewer zTcK (W2, Q3).

We also compared with SoMA on seven deformable-object scenarios using its official code and two-stage training procedure (Table 9; Fig. 15). GenTwin performs better under single-interaction training, while SoMA's original configuration targets joint learning across multiple interactions and may require manual hyperparameter tuning for this setting. Details are provided in Appendix C.4 and our response to Reviewer DvFt (WQ3).

3. Robustness to upstream errors and random seeds. As requested by Reviewer zTcK, we evaluated four types of controlled perturbations on one representative scenario from each object category: geometric priors, with three levels of mesh stretching and rotation; tracking trajectories, with three levels of Gaussian noise; part segmentation, with part mixing and splitting; and material recognition, with 20% stiffness-bound changes or substitutions with similar or substantially different materials. Results (Sec. 4.4; Fig. 10; Appendix D.5) show limited degradation even under the most severe tested perturbations, supporting robustness to these upstream errors. Details are provided in our response to Reviewer zTcK (W3, Q4-Q5).

We also evaluated five CMA-ES seeds per scenario on the same three cases, as suggested by Reviewer bT3Z. Table 16 reports metric means and standard deviations; Fig. 19 shows convergence curves with the final generations enlarged. The low standard deviations indicate consistent future prediction performance under the tested seed changes. Details are provided in Appendix D.6 and our response to Reviewer bT3Z (W2).

4. Implementation details and computational cost. To address concerns from Reviewers zTcK and DvFt, we clarified that PhysTwin and GenTwin use the official PhysTwin data as provided, and share the same preprocessed observations, initial geometry, 3D tracks, hand trajectories, and train/test splits on the 11 self-captured scenarios. Both use 20 CMA-ES generations and 200 Adam updates with the same learning rate. Appendices C.2-C.3 detail baseline preprocessing, controller settings, and optimization. Details are provided in our responses to Reviewers zTcK (Q3.2) and DvFt (WQ4-WQ5).

We also detailed the particle dynamics, contact and friction formulations, and numerical state updates in Appendix B.5, and clarified local and long-range spring rest-length initialization in Sec. 3.4, addressing Reviewer DvFt (WQ1-WQ2). Following Reviewer bT3Z's suggestion, we profiled runtime and CPU/GPU memory consumption for CMA-ES, refinement, and inference, along with simulation FPS, on three representative scenarios. Details are provided in our response to Reviewer bT3Z (Q1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.