- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses HOI reconstruction task by introducing dynamics-aware objectives (). From the initial hand and object estimates, the geometric features of the hand and the object are encoded via diffusion process and temporal self-attention. Visual grounding is applied with DINOv2 features and final HOI is reconstructed via diffusion-based interaction refinement network. The proposed method is trained and evaluated on HOT3d, HO3Dv2, and DexYCB, showing improvements on hand and object trajectory reconstruction.
Critical strengths
- Dynamics-aware HOI reconstruction is a logical and technically interesting direction, and the writing in the Introduction section convinces the motivation well.
- The proposed pipeline is sound. Geometry encoding, visual grounding, and interaction refinement seem convincing and are presented well.
- Strong comparative benefits are shown on HOT3D, especially acceleration and object ADD (Table 1).
- The ablation study in Table 3 provides impacts of each proposed term comprehensively, supporting the validity of their contributions.
Critical weaknesses
- Ablation and SOTA reports are conducted on the same evaluation split (HOT3D, HO3Dv2). (Table 2, Table 3). It is concerning that its strong comparative benefit might be attributed to overfitting to the target benchmark, and not be generalizable to the unseen domain. It seems that HOT3D and HO3Dv2 do not provide official train-val-test splits, but the authors could have split the original train split into their own train-val split.
- The comparative benefit might come from incorporating more datasets, rather than the proposed method itself. While WHOLE (Ye et al., 2026) is trained on HOT3D, the proposed framework is trained with 3 different datasets, using more data than the comparative baseline. It would be helpful to disentangle the architectural framework from the training data composition. For example, one could fairly compare the proposed framework with the baselines like WHOLE by training only on HOT3D.
- The distribution choice ablation in Table 4 does not fully validate their choice. Gaussian and Laplace distributions also have their own parameters to control sharpness (), similar to in Student-t distribution. For a fair comparison, parameters of each distribution should be fitted with the same principle.
- While the paper emphasizes the challenges of occlusion (L035, L042, L157, L419), the paper lacks stratified occlusion analysis of the proposed framework. It would be more convincing to compare the proposed method with the baselines under different levels of occlusion.
Questions for authors
- If the train-val-test splits are made and experimented, would the performance trend be different from Table 3?
- Is there a comparative benefit over the baselines (e.g., WHOLE) when the proposed framework is trained with the same training data composition?
- In Table 4, which distribution would be the best if their parameters are fitted with the same principle?
- How does the proposed method differ from the baselines under stratified occlusion analysis?
5 comments
Thank you for the response and the additional results. I read other reviewers' comments and also agree with the necessity of physical plausibility evaluation. Your additional table in "W1. Physical Plausibility Evaluation", reviewer 43wU partially addresses this issue, but could be extended to incorporate HaWoR or WHOLE. One could also conduct an object displacement evaluation using a simulation software [1, 2].
[1] Hanwen Jiang et al., Hand-Object Contact Consistency Reasoning for Human Grasps Generation, ICCV 2021
[2] Yufei Ye et al., G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis, CVPR 2024
Rebuttal Follow-up to Reviewer bpsv
Thank you for the follow-up. We conduct both additional evaluations as suggested.
| Method | Penetration Difference (mm) ↓ | Contact P / R / F1 (%) ↑ | Sim. Displacement (cm) ↓ | Stable Frames (%) ↑ |
|---|---|---|---|---|
| GT annotation | – | – | 1.79 | 59.7 |
| WiLoR + FoundationPose | 2.62 | 90.7 / 14.4 / 24.9 | 9.46 | 5.6 |
| HaWoR + FoundationPose | 2.62 | 96.5 / 18.8 / 31.4 | 9.76 | 4.3 |
| WiLoR + GoTrack | 1.38 | 91.2 / 34.0 / 49.5 | 8.01 | 12.3 |
| HaWoR + GoTrack | 0.91 | 97.3 / 48.5 / 64.7 | 7.20 | 17.7 |
| Ours (WiLoR + FoundationPose) | 0.08 | 96.4 / 60.0 / 74.0 | 7.10 | 19.0 |
| Ours (HaWoR + GoTrack) | 1.33 | 96.0 / 76.1 / 84.9 | 5.32 | 30.8 |
Extended Physical Plausibility Evaluation. Since WHOLE has not released its code or pretrained model, we additionally evaluate three combinations of off-the-shelf estimators: HaWoR + FoundationPose, WiLoR + GoTrack [3], and HaWoR + GoTrack. We also evaluate our framework with both WiLoR + FoundationPose and HaWoR + GoTrack initializations, following our previous contact and penetration evaluation protocol.
Simulated Object Displacement. Following [1, 2], we evaluate grasp stability in PyBullet by fixing the reconstructed hands and measuring object displacement under gravity on ground-truth holding frames. Lower displacement and a higher proportion of stable frames indicate better grasp stability.
Our method improves contact consistency and simulated grasp stability under both initializations. Notably, HaWoR + GoTrack provides stronger initial estimates than the WiLoR + FoundationPose combination used in our paper, and our framework achieves further improvements with these newer estimators. This demonstrates its flexibility and potential to benefit from future advances in pose estimation. We will include these evaluations and discuss the remaining limitations in the revision.
[1] Jiang et al., Hand-Object Contact Consistency Reasoning for Human Grasps Generation, ICCV 2021.
[2] Ye et al., G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis, CVPR 2024.
[3] Nguyen et al., GoTrack: Generic 6DoF Object Pose Refinement and Tracking, CVPR Workshops 2025.
Response to Reviewer bpsv
W1/Q1. Data Split and Model Selection. We follow WHOLE's HOT3D evaluation protocol to use held-out dynamic trajectories (object displacement > 5 cm) from annotated HOT3D-Clips as the test set. We further reserve ~10% of the remaining training clips for validation. All model selection and hyperparameter tuning are performed on the validation split without accessing test performance. Thus, Table 3 already follows a train/val/test protocol. We will clarify this in Sec. 4.1.
W2/Q2. Training Data Comparison. We clarify that our models are trained independently on each dataset without cross-dataset training. Specifically, same as WHOLE, our HOT3D model is trained exclusively on HOT3D training clips without using HO3D-v2 or DexYCB data. Therefore, the improvements cannot be attributed to additional training data.
W3/Q3. Distribution Comparison. For a fair comparsion with Student-t prior, we refit all other distributions using per-DoF maximum likelihood estimation (MLE) with the same per-DOF standardization and training settings on the HOT3D actuation data. Student-t achieves the lowest NLL and the best PA and [email protected], while remaining competitive on other metrics. We will update Table 4 accordingly in the revision.
W4/Q4. Occlusion-Stratified Evaluation. Based on the annotations provided by HOT3D dataset with occlusion ratio defined as 1 − visibility, hand and object results are stratified separately using their respective per-frame visibility. Results show that our method consistently outperforms the baselines across all occlusion levels.