acceptodds
Withdrawn from ICLR 2027

DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand-Object Interaction, 3D Reconstruction, Physics-Aware Learning, Diffusion Models, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bpsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses HOI reconstruction task by introducing dynamics-aware objectives (Ldyn,LactL_{dyn}, L_{act}). From the initial hand and object estimates, the geometric features of the hand and the object are encoded via diffusion process and temporal self-attention. Visual grounding is applied with DINOv2 features and final HOI is reconstructed via diffusion-based interaction refinement network. The proposed method is trained and evaluated on HOT3d, HO3Dv2, and DexYCB, showing improvements on hand and object trajectory reconstruction.

Critical strengths
  • Dynamics-aware HOI reconstruction is a logical and technically interesting direction, and the writing in the Introduction section convinces the motivation well.
  • The proposed pipeline is sound. Geometry encoding, visual grounding, and interaction refinement seem convincing and are presented well.
  • Strong comparative benefits are shown on HOT3D, especially acceleration and object ADD (Table 1).
  • The ablation study in Table 3 provides impacts of each proposed term comprehensively, supporting the validity of their contributions.
Critical weaknesses
  • Ablation and SOTA reports are conducted on the same evaluation split (HOT3D, HO3Dv2). (Table 2, Table 3). It is concerning that its strong comparative benefit might be attributed to overfitting to the target benchmark, and not be generalizable to the unseen domain. It seems that HOT3D and HO3Dv2 do not provide official train-val-test splits, but the authors could have split the original train split into their own train-val split.
  • The comparative benefit might come from incorporating more datasets, rather than the proposed method itself. While WHOLE (Ye et al., 2026) is trained on HOT3D, the proposed framework is trained with 3 different datasets, using more data than the comparative baseline. It would be helpful to disentangle the architectural framework from the training data composition. For example, one could fairly compare the proposed framework with the baselines like WHOLE by training only on HOT3D.
  • The distribution choice ablation in Table 4 does not fully validate their choice. Gaussian and Laplace distributions also have their own parameters to control sharpness (σ,b\sigma, b), similar to ν\nu in Student-t distribution. For a fair comparison, parameters of each distribution should be fitted with the same principle.
  • While the paper emphasizes the challenges of occlusion (L035, L042, L157, L419), the paper lacks stratified occlusion analysis of the proposed framework. It would be more convincing to compare the proposed method with the baselines under different levels of occlusion.
Questions for authors
  • If the train-val-test splits are made and experimented, would the performance trend be different from Table 3?
  • Is there a comparative benefit over the baselines (e.g., WHOLE) when the proposed framework is trained with the same training data composition?
  • In Table 4, which distribution would be the best if their parameters are fitted with the same principle?
  • How does the proposed method differ from the baselines under stratified occlusion analysis?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bpsv

W1/Q1. Data Split and Model Selection. We follow WHOLE's HOT3D evaluation protocol to use held-out dynamic trajectories (object displacement > 5 cm) from annotated HOT3D-Clips as the test set. We further reserve ~10% of the remaining training clips for validation. All model selection and hyperparameter tuning are performed on the validation split without accessing test performance. Thus, Table 3 already follows a train/val/test protocol. We will clarify this in Sec. 4.1.

W2/Q2. Training Data Comparison. We clarify that our models are trained independently on each dataset without cross-dataset training. Specifically, same as WHOLE, our HOT3D model is trained exclusively on HOT3D training clips without using HO3D-v2 or DexYCB data. Therefore, the improvements cannot be attributed to additional training data.

W3/Q3. Distribution Comparison. For a fair comparsion with Student-t prior, we refit all other distributions using per-DoF maximum likelihood estimation (MLE) with the same per-DOF standardization and training settings on the HOT3D actuation data. Student-t achieves the lowest NLL and the best PA and [email protected], while remaining competitive on other metrics. We will update Table 4 accordingly in the revision.

Prior (MLE)NLL ↓W ↓PA ↓ACCEL ↓[email protected] ↑[email protected] ↑
Gaussian1.425.304.491.7125.347.9
Laplace0.195.314.261.6925.249.2
Cauchy-0.875.144.201.6824.947.6
Student-t-1.595.264.151.6825.848.2

W4/Q4. Occlusion-Stratified Evaluation. Based on the annotations provided by HOT3D dataset with occlusion ratio defined as 1 − visibility, hand and object results are stratified separately using their respective per-frame visibility. Results show that our method consistently outperforms the baselines across all occlusion levels.

OcclusionWiLoR W / PAHaWoR W / PA ↓Ours W / PA ↓FoundationPose [email protected] / [email protected] ↑Ours [email protected] / [email protected] ↑
0–25%11.13 / 6.1911.50 / 7.995.08 / 3.9513.9 / 27.126.6 / 49.6
25–50%12.16 / 7.3411.20 / 9.265.71 / 4.778.2 / 18.925.1 / 45.0
≥50%16.51 / 8.4814.37 / 10.796.27 / 5.164.4 / 11.414.0 / 32.3
All11.74 / 6.5211.73 / 8.385.26 / 4.1512.8 / 25.325.8 / 48.2

Thank you for the response and the additional results. I read other reviewers' comments and also agree with the necessity of physical plausibility evaluation. Your additional table in "W1. Physical Plausibility Evaluation", reviewer 43wU partially addresses this issue, but could be extended to incorporate HaWoR or WHOLE. One could also conduct an object displacement evaluation using a simulation software [1, 2].

[1] Hanwen Jiang et al., Hand-Object Contact Consistency Reasoning for Human Grasps Generation, ICCV 2021

[2] Yufei Ye et al., G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis, CVPR 2024

AuthorsOpenReview ↗

Rebuttal Follow-up to Reviewer bpsv

Thank you for the follow-up. We conduct both additional evaluations as suggested.

MethodPenetration Difference (mm) ↓Contact P / R / F1 (%) ↑Sim. Displacement (cm) ↓Stable Frames (%) ↑
GT annotation––1.7959.7
WiLoR + FoundationPose2.6290.7 / 14.4 / 24.99.465.6
HaWoR + FoundationPose2.6296.5 / 18.8 / 31.49.764.3
WiLoR + GoTrack1.3891.2 / 34.0 / 49.58.0112.3
HaWoR + GoTrack0.9197.3 / 48.5 / 64.77.2017.7
Ours (WiLoR + FoundationPose)0.0896.4 / 60.0 / 74.07.1019.0
Ours (HaWoR + GoTrack)1.3396.0 / 76.1 / 84.95.3230.8

Extended Physical Plausibility Evaluation. Since WHOLE has not released its code or pretrained model, we additionally evaluate three combinations of off-the-shelf estimators: HaWoR + FoundationPose, WiLoR + GoTrack [3], and HaWoR + GoTrack. We also evaluate our framework with both WiLoR + FoundationPose and HaWoR + GoTrack initializations, following our previous contact and penetration evaluation protocol.

Simulated Object Displacement. Following [1, 2], we evaluate grasp stability in PyBullet by fixing the reconstructed hands and measuring object displacement under gravity on ground-truth holding frames. Lower displacement and a higher proportion of stable frames indicate better grasp stability.

Our method improves contact consistency and simulated grasp stability under both initializations. Notably, HaWoR + GoTrack provides stronger initial estimates than the WiLoR + FoundationPose combination used in our paper, and our framework achieves further improvements with these newer estimators. This demonstrates its flexibility and potential to benefit from future advances in pose estimation. We will include these evaluations and discuss the remaining limitations in the revision.

[1] Jiang et al., Hand-Object Contact Consistency Reasoning for Human Grasps Generation, ICCV 2021.

[2] Ye et al., G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis, CVPR 2024.

[3] Nguyen et al., GoTrack: Generic 6DoF Object Pose Refinement and Tracking, CVPR Workshops 2025.

Reviewer_43wUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DynamicHOI, a physics-aware framework for hand-object interaction (HOI) reconstruction. The framework combines (1) geometry-grounded diffusion refinement, (2) articulated inverse dynamics for hand dynamics, (3) Newton-Euler dynamics for object dynamics, and (4) coupling hand-object dynamics through contact-force transfer.

Given the monocular RGB videos and the object template, they first estimate hand parameters and object parameters with off-the-shelf estimators. The proposed diffusion-based refinement network refines the initial trajectory, capturing the geometric features, temporal information, visual grounding features, and spatial-temporal information. In addition to the diffusion objective L_{diff}, the refinement network is trained with hand/object dynamics L_{dyn} and their coupled mechanics L_{act}. L_{dyn} compares the hand generalized forces and object wrenches between the predicted trajectory and ground-truth trajectory. Lastly, L_{act} enforces that active hand actuation of the predicted trajectory follows the ground-truth distribution.

Experiments on egocentric and allocentric HOI datasets demonstrate that DynamicHOI outperforms prior HOI methods. Particularly large gain on the egocentric dataset comes from visual grounding in Tab. 2(a), and physics-aware objective in Tab. 3(a).

Critical strengths

While previous works only handle articulated inverse dynamics or Newton-Euler dynamics, DynamicHOI combines both for HOI reconstruction. In addition, the authors propose coupled HOI dynamics to model mutually consistent hand—object interaction. L_{dyn} and L_{act} provides consistent gains in Tab. 3(a). The proposed components are well motivated and validated with ablation study.

Critical weaknesses

Evaluation

  • Quantitative evaluation demonstrate that DynamicHOI outperforms prior HOI methods across various metrics. However, the metrics cannot evaluate physically plausibility.
  • Qualitative evaluation is limited. They provide visualization in Fig. 3, but it is hard to distinguish whether DynamicHOI performs better or not. Qualitative videos or visualization especially on contact region will be helpful.
Questions for authors
  1. Why L_{act} uses negative log-likelihood instead of distance between the predicted active hand actuation and ground-truth?
  2. During training, y is estimated by WiLoR and FoundationPose? If so, DynamicHOI cannot generalize on other hand and object pose estimator? How about starting from the pure noise during training and guide the diffusion process with y during test time?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 43wU

W1. Physical Plausibility Evaluation. We evaluate contact precision, recall, and F1, along with the absolute difference in mean penetration depth relative to ground-truth annotations on HOT3D. Our method improves contact F1 and reduces the penetration difference, indicating more accurate recovery of hand-object interactions.

MethodMean Penetration Difference (mm) ↓Contact P / R / F1 (%) ↑
WiLoR + FoundationPose2.6290.7 / 14.4 / 24.9
Ours0.0896.4 / 60.0 / 74.0

W2. Qualitative Visualization. We provide a supplementary video with comparisons on five HOT3D test clips, including contact-region close-ups with hand vertices color-coded by distance to the object. Representative frames will also be added to the revised paper.

Q1. Why NLL Instead of Actuation Distance? The Ldyn\mathcal{L}_{\text{dyn}} already aligns the hand and object dynamics with ground truth, thus constraining actuation distance won't bring to much information gain. The Student-t NLL instead regularizes the actuation distribution while tolerating legitimate high-actuation events. The experimental results below show that Student-t NLL improves hand reconstruction while maintaining comparable object reconstruction performance.

Lact\mathcal{L}_{\text{act}}PA ↓ACCEL ↓[email protected] ↑[email protected] ↑
L2 to GT actuation4.801.7825.148.4
Student-t NLL (ours)4.151.6825.848.2

Q2. Generalization to Other Estimators and Noise Initialization. Our framework refines initial estimates y\mathbf y from WiLoR and FoundationPose but can also generalize to other estimators. We replace them with HaWoR and GoTrack and retrain using their estimations. The results show consistent improvements over both alternative estimators.

MethodW ↓PA ↓ACCEL ↓[email protected] ↑[email protected] ↑
HaWoR11.738.3812.90––
GoTrack–––40.358.9
Ours (y\mathbf y = HaWoR + GoTrack)6.825.581.8445.866.7

As suggested, we also train from pure noise and guide inference with y\mathbf y by replacing the predicted clean state at every reverse step, effectively denoising y\mathbf y perturbed to the lowest noise level (SDEdit-style).

MethodW ↓PA ↓ACCEL ↓[email protected] ↑[email protected] ↑
Pure noise21.4316.572.678.726.0
Pure noise + test-time guidance11.586.432.1012.131.0
Ours (Full)5.264.151.6825.848.2

Pure-noise training performs poorly, while test-time guidance improves reconstruction but remains substantially worse than our approach. These results show that our framework benefits from learning to refine y\mathbf y during training, while remaining compatible with different initial pose estimators. We will add these experiments to the revision.

AuthorsOpenReview ↗

Rebuttal Follow-up to Reviewer 43wU

Dear Reviewer 43wU, thank you again for your time and constructive feedback. Please let us know if our responses and additional experiments have addressed your concerns, or if you have any further questions.

Thank you for the response and additional experiments. I also read other reviews and the corresponding responses. Although the initial responses cannot address the concerns, the follow-up experiments address the concerns, especially simulated experiments. Therefore, I want to raise my rating to 3/4, weak accept.

Reviewer_gwfKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DynamicHOI, a physics-aware framework for monocular hand-object interaction reconstruction. The method combines geometry-grounded diffusion refinement with hand and object dynamics supervision, and further introduces a probabilistic prior over coupled hand-object actuation to improve reconstruction accuracy and physical plausibility. Experiments on three HOI datasets and two downstream applications demonstrate the effectiveness of the proposed approach.

Critical strengths

Relevant Research Problem. Existing HOI reconstruction methods primarily focus on visual and geometric consistency, while the reconstructed trajectories may still exhibit physically implausible motion. This paper addresses this limitation by incorporating hand-object dynamics into trajectory reconstruction, which is a meaningful research direction.

Unified Framework. The proposed method integrates geometry-grounded diffusion refinement, hand-object dynamics supervision, and coupled actuation regularization into a unified framework. The physical constraints are applied only during training, introducing no additional physics-related computational overhead at inference.

Critical weaknesses

Dynamics vs. Temporal Regularization. The generalized forces and object wrenches are derived entirely from motion trajectories without independent force measurements. In particular, gravity cancels out in the object dynamics loss, making it closely related to inertia-weighted motion consistency. The additional benefits over simpler temporal constraints remain unclear.

Contact-Force Validity. The minimum-norm inverse grasp mapping distributes object wrenches across six fixed hand anchors without explicitly enforcing valid contact locations, friction constraints, or unilateral contact conditions. Consequently, the estimated forces may not represent physically feasible interactions, weakening the claim of physically consistent hand-object coupling.

Actuation Prior Contribution. Although Table 3 demonstrates the effectiveness of the actuation prior, it remains unclear whether the improvements arise specifically from hand-object dynamics coupling or simply from additional regularization. Comparisons with a hand-only actuation prior or a prior without the transferred contact-force term would better justify the proposed coupling mechanism.

Physical Consistency Evaluation. Although the paper reports acceleration errors and motion-derived force/torque comparisons, these evaluations do not directly establish physically plausible hand-object interactions. Quantitative metrics such as contact consistency and hand-object penetration would provide stronger evidence.

Questions for authors

Baseline Comparison. Could the authors clarify the advantages of the proposed dynamics supervision over conventional temporal constraints, such as acceleration consistency? A comparison with a simple acceleration-based loss would help justify the necessity of explicit dynamics modeling.

Impact of Contact Approximation. The inferred contact forces may not satisfy physically feasible contact conditions. Could the authors further discuss how these approximations affect the reliability of the coupled actuation prior?

Coupled Actuation Ablation. Table 3 demonstrates the benefits of the actuation prior, but does not isolate the contribution of hand-object coupling. Could the authors compare the proposed coupled actuation prior with a hand-only dynamics prior to clarify the benefits of the coupling mechanism?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gwfK

W1. Dynamics vs. Temporal Regularization. Since force annotations are scarce, we derive dynamics supervision from pose annotations. Despite gravity cancellation, our formulation captures mass-dependent inertia, pose-dependent rotational inertia, and gyroscopic effects (Eq. 8), providing physically structured constraints beyond acceleration penalties. Our following response to Q1 empirically validates this advantage.

Q1. Comparison with Acceleration-Based Supervision. We replace Ldyn\mathcal{L}_{\text{dyn}} with two L2 acceleration losses, both supervising object linear and angular accelerations, alongside hand joint-position or generalized-coordinate accelerations, respectively. Our dynamics supervision consistently outperforms both alternatives, validating its advantage over simpler temporal constraints.

LossPA ↓ACCEL ↓[email protected] ↑[email protected] ↑
Joint-position accel.4.842.6721.741.3
Generalized-coord. accel.4.423.6823.347.5
Ldyn\mathcal{L}_{\text{dyn}} (ours)4.151.6825.848.2

W2/Q2. Contact-Force Approximation. We acknowledge that minimum-norm mapping does not enforce contact locations, friction cones, or unilateral constraints. Rather than recovering exact contact forces, we use a tractable approximation for coupled dynamics regularization, consistently applied to both predicted and ground-truth trajectories. Its effectiveness is supported by W3/Q3. We will clarify this limitation in the revision.

W3/Q3. Contribution of Coupled Actuation. We compare against a hand-only prior by removing Jc,t⊤ftJ_{c,t}^\top\mathbf f_t and refitting the Student-t distribution on ground-truth hand generalized forces, with all other settings unchanged. The improvements confirm the contribution of hand-object coupling beyond additional regularization.

PriorPA ↓[email protected] ↑[email protected] ↑
Hand-only4.3821.443.7
Coupled (ours)4.1525.848.2

W4. Physical Consistency Evaluation. We evaluate contact precision / recall / F1 and the absolute difference in mean penetration depth relative to ground-truth annotations. Our method improves contact F1 from 24.9% to 74.0% and reduces the penetration difference from 2.62 to 0.08 mm, indicating more accurate recovery of hand-object interactions.

MethodMean Penetration Difference (mm) ↓Contact P / R / F1 (%) ↑
WiLoR + FoundationPose2.6290.7 / 14.4 / 24.9
Ours0.0896.4 / 60.0 / 74.0
Reviewer_gwfKOpenReview ↗

Response to authors

The rebuttal provides additional evidence for dynamics-based regularization. However, the original ablation shows that most improvements in reconstruction accuracy are achieved without the coupled actuation prior, although the probabilistic modeling of active actuation is highlighted as a key insight in the Introduction Section . Moreover, the additional physical consistency evaluation only compares the full model with WiLoR + FoundationPose. Without ablations on the physical consistency metrics, it remains unclear whether these improvements come from the proposed dynamics modeling or general trajectory refinement.

Overall, I think that the current experiments cannot strongly support the paper's main claims. Therefore, I will keep my original rating.

AuthorsOpenReview ↗

Rebuttal Follow-up to Reviewer gwfK

Thank you for the follow-up. We address both points with a new ablations and discussion.

Ablation on Physical Consistency. Following prior work on simulation-based grasp evaluation [1, 2], we evaluate different training objectives using simulated object displacement alongside the physical consistency metrics in W4. Specifically, we fix the reconstructed hand poses in a physics simulator and measure object displacement under gravity, where lower displacement indicates more stable grasps. We evaluate ground-truth holding frames, using Ldiff\mathcal{L}_{\text{diff}} alone as the general trajectory refinement baseline.

Training objectiveSim. Displacement (cm) ↓Penetration Difference (mm) ↓Contact P / R / F1 (%) ↑
WiLoR + FoundationPose (input)9.462.6290.7 / 14.4 / 24.9
L_diff8.821.5993.0 / 32.5 / 48.2
L_diff + L_act7.300.6494.7 / 47.1 / 62.9
L_diff + L_dyn7.220.3695.5 / 51.4 / 66.8
Full (all three losses)7.100.0896.4 / 60.0 / 74.0

Both physics-aware objectives improve contact consistency and simulated grasp stability beyond general trajectory refinement. Combining them achieves the best results across all metrics, supporting their complementary contributions.

Role of the Actuation Prior. The coupled actuation prior provides additional benefits beyond independent dynamics supervision. As shown in Table 3, adding L_act reduces hand PA from 4.22 to 4.15 mm and ACCEL from 2.78 to 1.68, while our new ablation shows an improvement in contact F1 from 66.8% to 74.0%. These results demonstrate that coupled actuation modeling provides a complementary physical constraint that improves motion smoothness and contact consistency. Although its gains are smaller than those of L_dyn, we highlight that this approach introduces a new perspective for reconstruction through force-aware modeling beyond conventional geometric and temporal constraints. We will clarify this contribution in the Introduction and experimental discussion.

[1] Jiang et al., "Hand-Object Contact Consistency Reasoning for Human Grasps Generation", ICCV 2021.

[2] Ye et al., "G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis", CVPR 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.