acceptodds
Withdrawn from ICLR 2027

Reconstructing Deformable Hand-Object Manipulation via Vision-Physics Inverse Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing dynamic hand-object interaction from visual observations is fundamental for enabling embodied agents to acquire dexterous manipulation skills. While existing methods have achieved promising results under rigid-body assumptions or template-based representations, they struggle to capture the complex, spatially varying deformations arising in interactions with deformable objects, especially in bimanual settings. In this paper, we present a novel vision-physics inverse modeling framework for reconstructing bimanual dynamic deformable hand-object interactions from multi-view videos. Our key insight is that object deformation is a direct physical response to hand-applied forces, which is implicitly encoded in temporal appearance variations across views. Building on this observation, our method jointly leverages geometry, appearance, and interaction cues from multi-view observations to support physically grounded non-rigid reconstruction. Specifically, we introduce a physics-aware deformation model that estimates both a time-varying force field and a time-invariant material field, allowing fine-grained deformations to be recovered via differentiable simulation. To facilitate evaluation, we further construct a real-world multi-view dataset of bimanual interactions with diverse deformable objects. Extensive experiments demonstrate that our method significantly outperforms prior approaches in both geometric accuracy and physical plausibility, particularly in challenging deformable interaction scenarios. The code and dataset will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand Pose Estimation, Non-rigid object Manipulation, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EiWMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reconstructs bimanual interactions with deformable objects from calibrated multi-view RGB video. It estimates hand and object geometry, then fits latent force and material fields through differentiable material-point simulation. The paper also introduces a 20-sequence dataset recorded with six cameras and evaluates reconstruction on ARCTIC and HMDO, with ablations that examine both geometry and penetration.

Critical strengths
  1. I find the joint treatment of hand motion, contact, and nonrigid object deformation well motivated. These quantities constrain one another under occlusion, so fitting them together is a sensible contribution to a difficult reconstruction problem.
  2. The object-surface results on HMDO are promising, and the paired ablations report penetration as well as geometry. Those checks are relevant to the intended use of the reconstructions and give some empirical support to the interaction-modeling approach.
Critical weaknesses
  1. Two parts of the physical formulation prevent me from assessing the method confidently. Appendix Eq. (9) is called Neo-Hookean but appears closer to a fixed-corotated energy. Eq. (4) uses unsigned distances for a penetration penalty, without explaining how interior and exterior points are distinguished. The implemented stress and collision objectives need to be clear because they define the proposed inverse-physics model.
  2. The evaluation is too narrow for me to judge how reliable the method is. Fig. 3 shows only two qualitative comparisons; the video adds examples, but offers little analysis of failure cases or performance across deformation types. The main HMDO comparison covers just five sequences (150 frames), selected because all methods returned valid outputs. The ten-sequence ablations are useful, but without broader reporting and comparisons using the same inputs, I cannot tell whether the gains hold beyond the selected cases.

Missing references: PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions (ICML 2026), AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation (SIGGRAPH 2026).

Questions for authors
  1. Which constitutive energy and stress law are actually implemented? Could you reconcile Appendix Eq. (9) with the term “Neo-Hookean” and clarify whether this is a labeling issue or a difference in the implemented model?
  2. How does Eq. (4) distinguish penetration from nearby external contact? Is a signed-distance, occupancy, or separate collision term used in the experiments?
  3. How many HMDO sequences did each method attempt and complete, and what observations did each receive? Please identify any comparison that holds the inputs fixed and explain how including failed cases affects the conclusions.
  4. Could you show reconstruction sequences covering the evaluated objects and deformation types, including difficult or failed cases, alongside per-sequence errors? This would help me judge whether the examples and average results are representative.
Reviewer_uG5vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reconstructs dynamic interactions between two hands and deformable objects from calibrated multi-view RGB videos. It first initializes and optimizes hand–object geometry, appearance, and rigid alignment, then jointly optimizes force fields, material fields, and hand poses through differentiable MPM. The authors also construct a six-view dataset containing 20 interaction sequences, 10 objects, and 1,000 selected frames.

On selected ARCTIC and HMDO sequences, the method reports the best object CD and F-scores, while its hand MPJPE remains worse than HOLD. The appendix includes a PhysTwin+DA3 comparison, contact metrics, force-field visualizations, and experiments using the new data to train IHOI.

Critical strengths

The task is valuable, and the system jointly reconstructs hands, deformable objects, and their interaction. The object results in Table 1, the matched-initialization ablations in Table 2, and the adapted PhysTwin comparison in Table 3 support different aspects of the system. Bimanual deformable interaction is a worthwhile setting beyond rigid-object HOI or object-only dynamics reconstruction.

The dataset has evidence of downstream value on an external benchmark. The authors explicitly state that the reconstructed annotations are not independent ground truth for evaluating their own method. In Table 5, adding the new data to IHOI reduces HMDO MPJPE from 25.32 to 18.10 and CD from 10.87 to 3.92, supporting its value as downstream supervision.

Critical weaknesses
  1. The CD and F-score evaluation definitions need to be checked.

    Section 5.2 reports CD in mm and F-scores at 5/10 mm thresholds, but does not define CD mathematically. If CD is the average of the two directional mean unsquared Euclidean nearest-neighbor distances, computed on the same point clouds with the same alignment, and precision/recall use those same distances and thresholds, then: $ F \geq 1-\frac{2,\mathrm{CD}}{\tau}. $

    Here, FF is on the 0–1 scale. Averaging frames and sequences with the same weights preserves this necessary condition. In Table 1, the ARCTIC CD of 1.11 mm requires an F-score of at least 55.60% at 5 mm, whereas the reported value is 52.14%. In Table 2, the variant without deformation has CD 2.16 mm, requiring at least 56.80% at 10 mm, whereas 54.80% is reported. Rounding does not explain these differences.

    This concern is conditional. Squared CD, different sampling or alignment, or different aggregation could invalidate the bound, so the available description does not establish a computational error. Appendix A.5 specifies frame-then-sequence averaging, and the different subsets used in Tables 1 and 2 are disclosed; different full-model results between those tables are not themselves inconsistent.

    Please provide the CD formula, units, point counts, alignment, aggregation rules, and per-frame precision/recall/CD values. If the definitions explain the discrepancy, this concern reduces to reporting clarity. Otherwise, the main results need to be recomputed under a consistent protocol.

  2. The contact constraints and simulation are not specified precisely enough.

    Equation (4) on page 6 penalizes distances between Gaussian centers below a threshold, without an inside/outside test or signed distance. It can impose short-range repulsion, but does not directly measure penetration depth. For example, nested surfaces can receive zero penalty when their separation exceeds the threshold. The attraction target τc\tau_c is unspecified, leaving its interaction with the 5 mm repulsion threshold unclear.

    Tables 2 and 4 show MSPD decreasing from 2.14 to 1.75 mm and PVR from 0.68% to 0.28%, which supports the practical value of the constraint. The independent inside/outside tests used to compute these metrics are needed to interpret the penetration improvement.

    Several simulation details also need reconciliation. Equation (9) in Appendix A.1 is called Neo-Hookean, but its energy has the fixed-corotated form; see Physics3D, Equation (8). Appendix A.9 specifies 20 steps per frame at a time step of 10−410^{-4}, totaling 0.002 simulation time units; the mapping to 30 FPS video and physical units is not explained. The main paper restricts forces to near-contact regions, whereas Appendix A.8 allows forces across the whole object for global deformation, without a clear selection rule. Effective latent parameters can use normalized time and distributed loads, but these choices must be reproducible.

  3. The benefit of physics remains confounded with strong initialization and added deformation freedom.

    The method uses calibrated multi-view input, strong object initialization, and per-sequence optimization, but the input and initialization adaptations for the main HOI baselines are unclear. Table 2 does contain a controlled comparison with the same sequences and initialization: enabling deformation reduces CD from 2.16 to 2.09 and improves F@5/10 by 2.75/3.16 percentage points. This supports allowing deformation, but does not establish that MPM outperforms a free nonrigid Gaussian deformation model under the same budget. The triplane versus particle-wise force comparison is qualitative.

    Table 3 includes PhysTwin, and the proposed method also reconstructs the hands, which is a practical advantage. However, PhysTwin uses DA3-estimated depth, so the gap also reflects depth noise and hand modeling. The five-sequence HMDO subset on which all methods produce valid outputs is disclosed; per-method failure counts and per-sequence results would make this comparison more informative.

    The most important addition is a nonphysical deformation control with matched views, initialization, and optimization budget. Please also specify the new dataset's object- or sequence-level train/test split, stage runtimes, and the relationship between the main paper's “50 frames” and Appendix A.5's “8 sequences, 240 frames” for ARCTIC. These details determine how much of the overall advantage can be attributed to the proposed physical design.

Questions for authors
  1. What are the complete CD formula, units, scaling, and point-cloud protocol? Can per-frame results reconcile the CD/F-score condition above? If CD is squared, how should its reported mm unit be interpreted?
  2. Does the implementation include an additional inside/outside test? How are MSPD and PVR computed, and how do τc\tau_c, the repulsion threshold, and Gaussian scales interact?
  3. Is the actual constitutive model fixed-corotated or Neo-Hookean? How do 20 simulation steps map to video time? When are forces restricted to the 2 cm contact neighborhood, and when are whole-object forces allowed?
  4. What is the gain from MPM over free nonrigid deformation with matched inputs, initialization, and budget? What are the baseline adaptation procedures, failure rules, and dataset splits?
  5. Can the authors provide a consistent list of evaluation sequences and frames, stage runtimes, and constraints on the material and force fields?
Reviewer_VNkiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reconstructs deformable bimanual hand–object interactions from multi-view RGB videos by jointly optimizing hand poses, time-varying forces, and a material field through differentiable Gaussian–MPM simulation.

Critical strengths

This paper addresses hand–deformable-object reconstruction without measured depth by jointly fitting hand poses and material fields to multi-view RGB observations. This is a practically useful setting beyond existing RGB-D formulations. Also, the controlled ablation shows that deformable optimization improves hand reconstruction, supporting the benefit of modeling deformation.

Critical weaknesses
  1. The distinction from closely related work is insufficiently established. The recent work PhysHanDI [1] also reconstructs dense hand–deformable-object interactions and refines hand poses through inverse physics, yet is neither discussed nor compared. Although RGB-only supervision is a meaningful difference, the additional benefit of the proposed force-field parameterization and joint optimization remains unclear.
  2. The contribution of material-field estimation is not isolated. The deformation ablation removes the entire physics stage rather than isolating material optimization. A fixed-material MPM control, retaining the same force parameterization, would clarify whether jointly estimating the material field improves reconstruction.

[1] Jihyun Lee, Changmin Lee, Donghwan Kim, and Tae-Kyun Kim. PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions. ICML, 2026. Paper.

Questions for authors
  1. Under comparable input conditions, what advantage does the learned force–material formulation offer over PhysHanDI’s contact-driven spring–mass model?
  2. With the force model fixed, does spatially varying stiffness improve reconstruction over a single learned stiffness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.