- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions. 4D-HOF learns a conditional flow matching model that transports foundation-model-derived initial estimations towards hand-object interaction distributions. Foundation models provide informative observations across diverse and challenging scenarios. During the generation process, test-time guidance improves image alignment, physical interaction and motion consistency.
Given an in-the-wild monocular RGB video, they leverage foundation models to obtain initial (1) hand pose, (2) object geometry, and (3) object pose. The initial hand and object pose are not aligned well due to the depth ambiguity, so they also use estimated depth map and build the initial hand—object interaction. 4D-HOF learns a conditional generative model that transports the initial hand—object states toward physically plausible interactions. This model is trained with the bridge matching objective (Eq. 4), 3D reconstruction loss (Eq. 5), interaction loss (Eq. 6), and regularization term (Eq.7). During the test-time, the generation process is guided by the silhouette (Eq. 10), interaction, and temporal guidance (Eq. 11).
Experiments on HOI datasets demonstrate strong performance and generalization ability of 4D-HOF with computational efficiency. 4D-HOF especially improve object accuracy and contact accuracy compared to the initial estimation.
Critical strengths
4D-HOF leverages vision foundation models, which enables feed-forward framework given an in-the-wild video. The initial outputs of various foundation models are not mutually aligned, so they align the outputs with estimated depth map to build rough HOI initialization. In addition, the conditional generative model refine the initial hand—object states as shown in Tab. 4. The generative nature of 4D-HOF also enables test-time guidance instead of a separate post-hoc optimization.
Critical weaknesses
4D-HOF transports the initial hand—object states estimated with vision foundation models toward physically plausible interactions. To train the model, they partially noise the ground-truth HOI states to emulate the imperfect initializations as described in Sec. A.2.2. How well the synthetic noise emulate the real distribution during the test time? How to design the synthetic noise including the hyper-parameters of the synthetic noise?
Questions for authors
Same as weaknesses.
Rebuttal 1/1
Thank you for the valuable comments!
Our synthetic perturbations are designed to span typical initialization errors while accounting for occasional severe failures. We have revised the perturbation sampling procedure in Appendix A.2.2 (lines 960–976) and added Figure S2 to compare the synthetic perturbation ranges with measured initializer errors. We selected and tuned the perturbation ranges and mixture probability empirically, assessing their suitability through error-coverage analysis and downstream refinement performance.
To quantify coverage, we evaluated empirical error distributions across HO-Cap, OakInk, SHOWMe, and HO3D. The synthetic perturbation bands encompass 99.8 – 100% of hand translation errors, 98.0 – 99.9% of object translation errors, and 89.2 – 95.1% of object rotation errors. Rather than enforcing an exact match to the real error distributions, our design prioritizes comprehensive support over realistic error bounds. We deliberately retain moderate rotation noise, as excessively large perturbations empirically degrade refinement quality. The significantly different hand MPJPE distribution on HO3D is consistent with this observation and supports balancing perturbation coverage with accurate refinement.