acceptodds
Withdrawn from ICLR 2027

HermiteTwin: Bicubic Hermite Shell Digital Twins for Contact-Rich Deformable Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While existing methods for reconstructing deformable object manipulation have made impressive progress, they typically represent a thin object with a single particle set that must serve two purposes at once: modeling bending, and resolving contact with the robot. To bridge this gap, we present \method, a framework that reconstructs a physical digital twin of a thin deformable object from multiview video of a robot manipulating it. Our key idea is to represent deformation and contact separately, each at its own resolution. The object is simulated as a bicubic Hermite shell, a coarse grid of elements whose nodal positions and derivatives define a smooth surface with Kirchhoff–Love bending, while the gripper acts through dense samples in material space that are captured, clamped, and released, and whose attachment forces reach the shell through the Hermite basis. This decoupling keeps the physical state compact while contact stays fine-grained. We identify effective material and contact parameters with CMA-ES followed by adjoint-based refinement, and drive a reconstructed Gaussian appearance model with the simulated shell. In experiments on real cloth and foam manipulation, \method outperforms the state-of-the-art spring–mass twin across reconstruction, resimulation, and future prediction.

Primary area:
optimization
Keywords:
Hermite Element, Gaussian Splitting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4eRYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HermiteTwin reconstructs a physical digital twin of a thin deformable object from calibrated multiview videos and recorded robot interactions. Its central idea is to decouple deformation and grasp resolution: a coarse bicubic Hermite shell models membrane and bending behavior, while dense material-space samples implement capture, clamp, and release through attachment constraints. Effective material and contact parameters are identified using CMA-ES followed by adjoint-based refinement. The simulated shell then drives a reconstructed Gaussian appearance model. Experiments on ten Deform360 interactions involving a red cloth and a white foam sheet show improved reconstruction, resimulation, and continuation prediction compared with PhysTwin and GS-Dynamics. Additional synthetic scenes demonstrate qualitative interactions with new actions and objects.

Critical strengths
  • The modeling choices are well motivated for thin deformable objects. Separating a compact shell state from dense grasp samples addresses the different resolution requirements of deformation and contact. Explicit, independently adjustable bending stiffness is appropriate for the cloth and foam examples. The attachment energy is also straightforward to integrate into the implicit solver: for fixed active samples and targets, it is quadratic in the shell state.

  • The reported improvements over the selected baselines are meaningful. Relative to PhysTwin, reconstruction CD decreases from 61.89 to 42.05 mm², and future-prediction CD decreases from 120.40 to 67.75 mm²—approximately 32% and 44% reductions, respectively. Silhouette and image metrics also improve. The qualitative examples illustrate better recovery of grasp-induced folds and preservation of the foam sheet’s shape.

Critical weaknesses
  • Limited scale and diversity of the evaluation data. The experiments use only 10 interaction sequences from Deform360, involving just two objects: a red cloth and a white foam sheet. This limited coverage does not sufficiently demonstrate the method’s generalization ability. Evaluation on additional datasets or a broader set of objects and interactions from Deform360 would strengthen the empirical evidence.

  • Limited baseline comparisons. The evaluation would benefit from comparisons with more recent baselines, such as SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation, or stronger spring–mass methods, such as NeuSpring: Neural Spring Fields for Reconstruction and Simulation of Deformable Objects from Videos. Other relevant, stronger baselines that are easier to reproduce would also be suitable.

Questions for authors
  • Cross-interaction transfer: Can the parameters identified from one interaction be held fixed to predict another real interaction involving the same object? How much do the identified material and contact parameters vary across recordings of the same object?

  • Additional experiments: As suggested in Critical Weaknesses, could the authors evaluate the method on more diverse data and compare it against stronger baselines?

Reviewer_uZihOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HermiteTwin aims to tackle the difficulty of capturing thin-object bending and localized robot grasps with a compact physical model. It proposes a framework that decouples deformation and grasp resolution by representing thin-object mechanics with a coarse bicubic Hermite shell and grasp capture, clamping, and release with dense material-space grasp samples whose attachment forces are transferred to the shell through the Hermite basis. Effective material and contact parameters are identified using CMA-ES followed by adjoint-based Adam refinement, while the simulated shell transports a separately reconstructed Gaussian appearance model. Experiments on ten cloth and foam-sheet interactions show improved reconstruction and resimulation over PhysTwin and GS-Dynamics, with better known-control future prediction on eight cloth interactions.

Critical strengths
  1. The separation of a compact Hermite shell from dense material-space grasp samples is well motivated for contact-driven thin-sheet manipulation. The paper is logically clear and easy to follow, with a coherent pipeline, useful implementation details in the appendix, informative visual comparisons, and explicit discussion of its limitations.

  2. The experimental protocol is reasonably designed, with shared observations, evaluation locations, and Gaussian appearance supporting comparison between simulators. Reported improvements are consistent across aggregate reconstruction and held-out continuation metrics. The modeling accuracy gains look promising.

Critical weaknesses
  1. The experimental coverage is narrow. Ten interactions involving only red cloth and a foam sheet provide limited object and material diversity. The selection criteria are unclear, including why other Deform360 objects compatible with the rectangular-sheet assumption were excluded. It is difficult to assess performance across more varied materials and deformation patterns within the method’s intended scope.

  2. The existing analyses only partially isolate the contribution. The grasp-transfer and friction-only studies are useful diagnostics, but broader quantitative ablations would strengthen attribution, e.g., a matched-grasp smooth-shell comparison, resolution and sampling-density controls, and an optimization-stage ablation would clarify which design choices drive the gains.

  3. The computational trade-off is insufficiently characterized. Fitting ten interactions takes approximately 6.5 hours on an RTX 5090, versus 5 hours for PhysTwin. A breakdown of forward simulation and adjoint costs, together with runtime scaling, would clarify the practical cost of the accuracy gains.

  4. Cross-trajectory parameter consistency remains unexamined despite its practical relevance and the cost of repeated identification. While joint multi-interaction identification is mentioned in limitation session and left to future work, the independently fitted material parameters could already be compared across trajectories involving the same physical sheet. Reporting these may help assess its potential.

Questions for authors
  1. How were the ten interactions selected from Deform360? Were other objects compatible with the rectangular-sheet assumption considered, and what limited their inclusion?

  2. If eight red-cloth interactions involve the same physical sheet, how consistent are the independently recovered membrane and bending stiffnesses?

  3. What are the main runtime bottlenecks, distinguishing outer search and refinement from inner simulation costs? How much accuracy does adjoint-based Adam add over the best CMA-ES candidate, and does this gain persist under a matched compute budget? As a possible future direction, could agent-guided initialization or parameter search reduce identification cost?

Reviewer_aVvyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HermiteTwin, a framework based on PhysTwin for simulating thin deformable objects using hermite shell physics instead of spring-mass physics and a new grasping mechanism. Experiments on real cloth and foam manipulation show better accuracy than prior SOTA.

Critical strengths

This paper tackles an important problem on how to simulate physically accurate Gaussian-based digital twins for a specific class of deformable objects. From the results provided, it seem to provide accuracy gain over prior SOTA.

Critical weaknesses
  1. Hermite shell choice is unclear: Spring-mass is a relatively simple simulation approach, so it is not surprising that a more sophiscated methods may perform better. Why choose a hermit shell over other existing cloth simulation methods? I would like authors to better justify their design choices with quantitative evidence.

  2. Limited problem scope: PhysTwin can model a broader range of deformable objects, while this paper seems to focus only on thin deformable objects. While I understand manipulation cloth is a popular task in robotics, this still seems like a significant reduction in generality. I would like the author to discuss whether this approach can be extended to other kinds of deformable objects.

  3. Baseline change: In 4.1, it seems that the authors trained PhysTwin with only chamfer loss, but looking at the publicly available source code of PhysTwin, that is not the way PhysTwin is trained. I am not sure why the authors changed this for PhysTwin, and whether this affects baseline PhysTwin's accuracy. I would like to see a comparison between PhysTwin trained with its original loss and with the modified loss, using the same dataset and training budget. Otherwise, I am not convinced that the reported results represent a faithful PhysTwin baseline.

  4. Insufficient evaluation and ablation studies. This paper is significantly lacking in results to support author’s design choices.

  • The paper uses a fixed 20×20 Hermite grid and 2 mm grasp sample spacing, but does not justify these choices. In particular, given that the gripper contact surface is much larger than 2 mm, it is unclear why such dense sampling is necessary. Would 5 mm or 10 mm spacing give similar results? Similarly, it is unclear whether the 20×20 grid is sufficient for larger objects or more complex folds. Lastly, while 4.4 provides a graph-transfer ablation, I do not find a breakdown of the benefit of the hermite shell physics and grasp model.

  • No simulation latency is provided. For interactive robotic applications, latency can be just as important as accuracy. The paper only provides fitting time, but not how fast the actual simulation runs. Furthermore, how does HermiteTwin scale with object size, grid size, and grasp sampling granularity? Without these results, it is hard to understand the practical cost of the proposed design.

  • The ablation in 4.4 is missing some important comparisons. If I understand correctly, PhysTwin + our grasp essentially uses mass nodes as grasp samples, but looking at the figure 4, PhysTwin’s mass nodes are more sparse then the HermiteTwin’s 21000 samples. So is the failure caused by the sparsity of the mass nodes in PhysTwin? I’d like the author to run an experiment where they have a much denser spring-mass graph with a similar number of grasp candidates and see if PhysTwin still fails. If not, would Hermite-shell physics outperform spring-mass physics?

  • I also have concerns about the 3D evaluation metrics used in this paper. Currently only chamfer distance is reported while PhysTwin they also included tracking error. Chamfer distance does not consider point identity so two objects can have similar shapes but very different material motion. Since the authors already seem to have tracked points (Appendix B), why is tracking error not included in the evaluation?

Questions for authors

See weakness

Reviewer_vmxWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HermiteTwin, a physics-based digital twin method for thin-shell deformable objects. At the state level, the method uses bicubic Hermite shells to represent smooth continuous geometry and Kirchhoff–Love bending energy; at the contact level, adhesion forces are applied to the shell through densely sampled material-space points via Hermite basis functions, thereby decoupling state resolution from contact resolution. Parameter identification is carried out using CMA-ES for global search combined with the adjoint method and Adam, and the simulated shell drives 3D Gaussian appearance rendering. On 10 real interaction sequences from the Deform360 dataset, the method outperforms PhysTwin and GS-Dynamics in both re-simulation and future prediction metrics.

Critical strengths

The decoupling of state and contact resolutions is well designed. By combining continuous Hermite shells with dense material-space sampling, the method preserves fine contact resolution while keeping the degrees of freedom at a compact scale of 5,292, effectively avoiding mesh proliferation.

In re-simulation of real robot interactions and long-horizon future prediction, the method significantly outperforms the compared baselines in both geometric and image-rendering metrics, and the qualitative reconstructions more completely retain fold/crease features.

The technical analysis is honest and rigorous: it not only analyzes in Appendix D why frictional contact causes the adjoint-method gradients to fail, but also, in the ablations, transfers the grasping mechanism to PhysTwin to demonstrate the benefit of the shell geometric representation itself.

The system engineering pipeline is complete, covering the full digital twin flow from multi-view input fitting, two-stage inverse parameter identification, to dynamics driven by unseen actions.

Critical weaknesses

The experiments are conducted only on red cloth and foam sheets within a single rectangular parameter domain across 10 sequences, and the foam sheets do not include temporal prediction. This creates a gap between the validation scenarios and the broad claims of deformable digital twinning made in the title and abstract, and there is no validation on non-rectangular cut patterns or anisotropic materials.

The unseen-action-driven results are purely simulation-rendered without real-world ground-truth comparisons. At the same time, future prediction still relies on the robot arm motions already present in the recorded videos. Because the inverted parameters have effective-value properties that mutually compensate, it is difficult to verify that the method has learned transferable real physics.

The comparison baselines are relatively weak: only mass-spring models that are prone to bending locking and learning-based networks that tend to diverge over long horizons are compared. There is no baseline based on triangular-mesh discrete shells or finite-element thin shells. The grasping-transfer ablation is also limited to a single-sequence qualitative demonstration, so the attribution of the gains is insufficient.

The main tables report only a single mean value, lacking per-sequence metrics, standard deviations, or confidence intervals. Moreover, since the parameter inversion relies on stochastic CMA-ES, no stability evidence from multi-seed repeated experiments is provided.

Ablation of key hyperparameters and modules is insufficient. The paper consistently uses a 20×2020\times20 grid, with no independent quantitative ablations for grid-resolution sensitivity, removal of the bending energy term, or the individual contributions of the two-stage optimizers.

The fairness of the comparison protocol is questionable. PhysTwin’s native loss function is replaced and its optimization budget is extremely small, while GS-Dynamics naturally accumulates error in unsupervised long-sequence rollouts. Thus, all baselines may not have been given the most favorable conditions for convergence.

Questions for authors

Could the authors supplement Table 1 and Table 2 with per-sequence metrics, variances, and stability data across multiple CMA-ES runs?

For the same material, can the fixed physical parameters identified on sequence A be directly transferred to sequence B for quantitative re-simulation tests?

For the same inversion task, how does the framework compare to the classic triangular-mesh discrete shells (Discrete Shells)?

Can the grasping-transfer ablation (Appendix C) be extended to all test sequences with a complete table of quantitative metrics?

What specific impact does grid resolution (e.g., changing to 10×1010\times10 or 40×4040\times40) have on computational time and simulation accuracy?

In long-horizon future prediction, what deformations are observed in the typical failure cases that produce the largest geometric errors?

Is the current limited optimization budget (4 generations of CMA-ES + 10 Adam steps) sufficient to guarantee adequate convergence for all baseline models?

Expectations for Rebuttal & Score Revision: The current score is primarily constrained by the limited empirical scale and missing validations. If the authors can convincingly address the points above—particularly by providing quantitative cross-sequence generalization results (Q2), per-sequence metrics with variance/significance statistics (Q1), and either a comparison against classical discrete/FEM shells or a full-dataset quantitative grasp transfer ablation (Q3/Q4)—I would be very happy to raise my score to Weak Accept during the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.