acceptodds
Withdrawn from ICLR 2027

4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models. Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc optimization after reconstruction, we directly steer the evolving generative states using physical interaction constraints and observed 2D evidence, allowing the reconstruction to be refined as part of the generative process itself. By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging in-the-wild scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand-object interaction, 3D/4D reconstruction, Motion processing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DKvCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions. 4D-HOF learns a conditional flow matching model that transports foundation-model-derived initial estimations towards hand-object interaction distributions. Foundation models provide informative observations across diverse and challenging scenarios. During the generation process, test-time guidance improves image alignment, physical interaction and motion consistency.

Given an in-the-wild monocular RGB video, they leverage foundation models to obtain initial (1) hand pose, (2) object geometry, and (3) object pose. The initial hand and object pose are not aligned well due to the depth ambiguity, so they also use estimated depth map and build the initial hand—object interaction. 4D-HOF learns a conditional generative model that transports the initial hand—object states toward physically plausible interactions. This model is trained with the bridge matching objective (Eq. 4), 3D reconstruction loss (Eq. 5), interaction loss (Eq. 6), and regularization term (Eq.7). During the test-time, the generation process is guided by the silhouette (Eq. 10), interaction, and temporal guidance (Eq. 11).

Experiments on HOI datasets demonstrate strong performance and generalization ability of 4D-HOF with computational efficiency. 4D-HOF especially improve object accuracy and contact accuracy compared to the initial estimation.

Critical strengths

4D-HOF leverages vision foundation models, which enables feed-forward framework given an in-the-wild video. The initial outputs of various foundation models are not mutually aligned, so they align the outputs with estimated depth map to build rough HOI initialization. In addition, the conditional generative model refine the initial hand—object states as shown in Tab. 4. The generative nature of 4D-HOF also enables test-time guidance instead of a separate post-hoc optimization.

Critical weaknesses

4D-HOF transports the initial hand—object states estimated with vision foundation models toward physically plausible interactions. To train the model, they partially noise the ground-truth HOI states to emulate the imperfect initializations as described in Sec. A.2.2. How well the synthetic noise emulate the real distribution during the test time? How to design the synthetic noise including the hyper-parameters of the synthetic noise?

Questions for authors

Same as weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal 1/1

Thank you for the valuable comments!

[W1] 4D-HOF transports the initial hand—object states estimated with vision foundation models toward physically plausible interactions. To train the model, they partially noise the ground-truth HOI states to emulate the imperfect initializations as described in Sec. A.2.2. How well the synthetic noise emulate the real distribution during the test time? How to design the synthetic noise including the hyper-parameters of the synthetic noise?

Our synthetic perturbations are designed to span typical initialization errors while accounting for occasional severe failures. We have revised the perturbation sampling procedure in Appendix A.2.2 (lines 960–976) and added Figure S2 to compare the synthetic perturbation ranges with measured initializer errors. We selected and tuned the perturbation ranges and mixture probability empirically, assessing their suitability through error-coverage analysis and downstream refinement performance.

To quantify coverage, we evaluated empirical error distributions across HO-Cap, OakInk, SHOWMe, and HO3D. The synthetic perturbation bands encompass 99.8 – 100% of hand translation errors, 98.0 – 99.9% of object translation errors, and 89.2 – 95.1% of object rotation errors. Rather than enforcing an exact match to the real error distributions, our design prioritizes comprehensive support over realistic error bounds. We deliberately retain moderate rotation noise, as excessively large perturbations empirically degrade refinement quality. The significantly different hand MPJPE distribution on HO3D is consistent with this observation and supports balancing perturbation coverage with accurate refinement.

Reviewer_g5NrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes 4D-HOF, a method for reconstructing 4D interactions between a single hand and an object from monocular video. Starting from coarse estimates provided by foundation models, the method uses conditional bridge matching, conditioned on image features and 3D geometric features, to refine the hand and object poses and their spatial alignment. It further incorporates test-time guidance to improve reconstruction performance. The method is evaluated on HO3D-v3 and DexYCB.

Critical strengths

Reconstruction Quality and Efficiency. The method substantially improves reconstruction quality while maintaining high computational efficiency. On HO3D-v3, it reduces the hand-relative Chamfer distance by approximately 62.2% compared with the strongest baseline and achieves a contact accuracy of 97.27%. It also offers a substantial runtime advantage over the baselines.

Generalization and Ablation Studies. The experimental evaluation is relatively comprehensive. The authors conduct cross-dataset evaluation using the same model and present qualitative results on in-the-wild data, providing evidence of the model’s generalization ability. Multiple ablation studies also examine the contributions of temporal modeling, 3D structural encoding, and test-time guidance.

Critical weaknesses

Limited methodological novelty. CARI4D has already explored foundation-model initialization followed by learned temporal pose refinement, while CHOIR employs Flow Matching to correct hand-object relative placement. Although 4D-HOF extends generative refinement to complete 4D hand-object states and incorporates test-time guidance, its overall methodology remains closely related to existing approaches. The paper does not sufficiently establish the distinct advantages of its conditional bridge formulation.

Missing Critical Baselines to Justify Bridge Matching. The current ablations validate individual components but do not directly establish the necessity of Conditional Bridge Matching. Two important comparisons are missing: (1) Gaussian-initialized conditional Flow Matching with the same conditioning signals, and (2) direct residual regression using the same initialization, backbone, and conditioning. These comparisons would help clarify whether the reported gains stem from structured initialization or generative refinement, and provide stronger justification for the proposed bridge-based design.

Potential Hand-Object Interpenetration. In the supplementary demo video (around 4:25), the reconstructed hand appears to penetrate the headphone headband from multiple viewpoints. This suggests that the proposed interaction guidance may still struggle to maintain physically plausible contacts in challenging in-the-wild scenarios, despite the improvements reported in the quantitative interaction metrics.

Questions for authors

Distinction from Prior Work. Given that CARI4D has explored learned pose refinement from foundation-model initialization and CHOIR has employed Flow Matching for hand-object interaction refinement, could the authors clarify the key methodological novelty of 4D-HOF beyond extending these ideas to full 4D hand-object states?

Alternative Refinement Baselines. Could the authors provide comparisons with (1) Gaussian-initialized conditional Flow Matching using the same conditioning signals and (2) direct residual regression using the same initialization, backbone, and conditioning? These baselines would help clarify whether the improvements arise from informative initialization, learned refinement, or the proposed bridge formulation.

4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for the valuable comments!

[W1] Limited methodological novelty. CARI4D has already explored foundation-model initialization followed by learned temporal pose refinement, while CHOIR employs Flow Matching to correct hand-object relative placement. Although 4D-HOF extends generative refinement to complete 4D hand-object states and incorporates test-time guidance, its overall methodology remains closely related to existing approaches. The paper does not sufficiently establish the distinct advantages of its conditional bridge formulation.

CARI4D and CHOIR both use three stages. They first initialize with foundation models, then apply learned refinement, and finally perform post-hoc numerical optimization. In the learned stage, CARI4D, which does not require fine-grained, finger-level object interaction, refines human-object states via a render-and-compare-based 2D module, whereas CHOIR employs flow matching solely to estimate a 1D scalar depth correction along the camera ray.

4D-HOF operates in two stages. First, foundation models generate a coarse hand–object trajectory that serves as the source distribution for our 4D conditional bridge model. The bridge then jointly refines hand and object poses over time, our ablation studies confirm that using this initialization as the source is critical. By applying test-time guidance directly to intermediate states during transport, our framework eliminates the need for a separate post-hoc optimization step. As shown in Table 3, guided bridge refinement requires only about one minute for a 200-frame sequence, compared to roughly 19 minutes for CHOIR’s optimization stage.

[W2] Missing Critical Baselines to Justify Bridge Matching. The current ablations validate individual components but do not directly establish the necessity of Conditional Bridge Matching. Two important comparisons are missing: (1) Gaussian-initialized conditional Flow Matching with the same conditioning signals, and (2) direct residual regression using the same initialization, backbone, and conditioning. These comparisons would help clarify whether the reported gains stem from structured initialization or generative refinement, and provide stronger justification for the proposed bridge-based design.

To address this concern, we conduct two additional ablations.

First, we replace the structured transport source with Gaussian noise while retaining the coarse initialization as conditioning. This variant exhibits a substantial performance drop. We attribute this degradation to the additional burden of constructing pose parameters from conditional input while also refining hand–object interactions. Using the coarse estimate as the transport source provides meaningful pose structure from the outset, allowing the learned dynamics to focus on interaction refinement.

Second, we retain the same network architecture but replace bridge matching with direct regression. This variant also performs worse than our full model across all reported metrics. Direct regression predicts the refined output in a single step, whereas bridge matching represents refinement through a time-dependent vector field integrated over multiple steps. The results suggest that this formulation better captures the complex, nonlinear refinement of hand–object interactions. Together, these ablations provide more direct evidence for the benefits of bridge matching formulation which takes the structured initialization as the transport source beyond using a learned model alone. We have incorporated these experiments in our revised PDF (lines 991-997).

VariantCD_hIVCAJMMPJPE
Gaussian74.1419.5488.6418.4337.24
Regression8.9315.2995.982.575.78
4D-HOF5.9813.4497.271.614.40
AuthorsOpenReview ↗

Rebuttal 2/2

[W3] Potential Hand-Object Interpenetration. In the supplementary demo video (around 4:25), the reconstructed hand appears to penetrate the headphone headband from multiple viewpoints. This suggests that the proposed interaction guidance may still struggle to maintain physically plausible contacts in challenging in-the-wild scenarios, despite the improvements reported in the quantitative interaction metrics.

Alongside achieving state-of-the-art performance on standard benchmarks, our demo includes the challenging headphone sequence to test generalization to unconventional manipulations. Lifting the headband using only the middle and ring fingertips is an out-of-distribution grasp pattern rarely represented in existing training data, which poses a natural challenge for the learned prior. Because our lightweight guidance is intentionally applied late in the transport process to preserve the prior's natural motion dynamics, residual contact offsets can remain when the initial trajectory reflects an unseen grasp type. Overall, however, our interaction guidance proves highly effective: on HO3D-v3, it reduces IV from 16.00 to 13.44 and boosts CA from 95.07% to 97.27% (Table 7). Expanding future HOI datasets to cover a wider spectrum of fine-grained grasp patterns will further enhance the learned prior on such specialized interactions.

[Q1] Distinction from Prior Work. Given that CARI4D has explored learned pose refinement from foundation-model initialization and CHOIR has employed Flow Matching for hand-object interaction refinement, could the authors clarify the key methodological novelty of 4D-HOF beyond extending these ideas to full 4D hand-object states?

4D-HOF is not merely an extension to 4D states, but a paradigm shift away from the three-stage pipelines (initialization → learned refinement → post-hoc optimization) used by CARI4D and CHOIR. While CARI4D relies on 2D rendering-and-compare and CHOIR limits generative flow matching strictly to a 1D scalar camera-ray translation, 4D-HOF introduces a unified 4D conditional bridge matching framework directly linking coarse foundation-model trajectories to plausible 4D hand-object states. This formulation models the complex transport across time, capturing fine-grained hand-object interaction within a single vector field. Crucially, enforcing physical constraints via in-flight test-time guidance on intermediate bridge states eliminates the need for separate post-hoc numerical optimization, yielding substantial efficiency gains (Table 3) alongside superior temporal-physical consistency.

[Q2] Alternative Refinement Baselines. Could the authors provide comparisons with (1) Gaussian-initialized conditional Flow Matching using the same conditioning signals and (2) direct residual regression using the same initialization, backbone, and conditioning? These baselines would help clarify whether the improvements arise from informative initialization, learned refinement, or the proposed bridge formulation.

We have evaluated both variants (detailed in our response to [W2] and added to Lines 991–997 of the revision PDF). These results clearly disentangle the sources of performance gain: (1) using Gaussian noise as the transport source severely degrades accuracy, confirming that setting the coarse foundation trajectory as the bridge source is essential to allow the network to focus on interaction refinement, and (2) single-step direct regression lags behind our model across all metrics, demonstrating that multi-step vector-field integration via bridge matching can help resolve complex, non-linear hand–object dynamics.

Reviewer_g5NrOpenReview ↗

Response to Authors

In terms of the overall pipeline, a key difference is that 4D-HOF integrates constraint-based refinement into the learned refinement stage as test-time guidance, reducing the three-stage pipeline to two stages. However, the underlying framework still follows a similar paradigm of foundation-model initialization, learned refinement, and constraint-based refinement, as explored in CARI4D and CHOIR.

Reviewer_g5NrOpenReview ↗

Response to Authors

I appreciate the authors' clarification regarding the challenging grasp pattern. However, the headphone sequence is featured in the teaser image to demonstrate the method's generalization ability, yet the supplementary video reveals noticeable hand-object interpenetration in 3D views. This raises concerns about the effectiveness of the proposed interaction guidance in challenging in-the-wild scenarios.

Reviewer_19BkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reconstruction of hands and rigid objects from monocular RGB videos. The authors first combine vision foundation models to obtain hand parameters, object meshes, per-frame object poses, depth, and camera information. Hand translation recovery, object keyframe selection, and enhanced pose estimation are used to construct coarse hand-object interaction sequences. A conditional generative bridge with temporal modeling then jointly refines the hand and object states. The model is conditioned on image features and hand-object point-cloud features, with silhouette, contact, penetration, and temporal smoothness guidance applied during the later stages of inference.

Critical strengths
  1. When the hand, object, and scene are estimated independently by different models, plausible local geometry does not necessarily yield correct interactions in a common coordinate frame. The paper addresses this inconsistency through translation recovery, spatial encoding, and joint state refinement. The method is well aligned with its motivation.

  2. The improvements in interaction consistency are supported by clear quantitative evidence.

Critical weaknesses
  1. The paper presents generative transport from structured initialization as its main innovation. However, the existing ablations primarily compare configurations with and without the learned model, or with and without individual model components. These comparisons demonstrate the effectiveness of learning, but do not establish whether the performance gains specifically require generative bridge modeling.

  2. The meaning of “feed-forward” should be clarified: does it refer to avoiding prolonged per-sequence fitting, or to requiring only a single network forward pass? The current description may lead readers to underestimate the iterative computation and gradient computation involved during inference.

  3. The paper attributes smooth trajectories to the learned interaction prior and states that they are obtained “without explicit smoothness regularization.” However, Eq. (11) explicitly includes velocity and acceleration penalties, and the full model uses this guidance. Table 7 shows that removing temporal guidance increases JM from 1.61 to 1.91. Therefore, the full-model results alone cannot establish that smoothness comes entirely from the learned prior.

  4. Similarly, the claim of improved physical plausibility should reflect the trade-offs across metrics. In Table 4, “VFM only” has an IV of 11.06, whereas the full model has an IV of 13.44. Although the full model substantially improves CDh and CA, it does not improve all physical metrics relative to the initialization.

Questions for authors

What would the results be if the model started from Gaussian noise while using the coarse initialization as a condition? How much of the improvement comes from the structured source distribution itself?

2 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for the valuable comments!

[W1] The paper presents generative transport from structured initialization as its main innovation. However, the existing ablations primarily compare configurations with and without the learned model, or with and without individual model components. These comparisons demonstrate the effectiveness of learning, but do not establish whether the performance gains specifically require generative bridge modeling.

To isolate the sources of performance gain, we conduct additional ablation studies comparing two key variants: (1) using standard Gaussian noise as the transport source, and (2) direct regression training. Both variants underperform our full model, with the Gaussian source exhibiting a particularly large performance drop. This underscores the importance of structured initialization as translation input and validates our bridge matching formulation. We have incorporated this discussion in our revised PDF (lines 991-997).

VariantCD_hIVCAJMMPJPE
Gaussian74.1419.5488.6418.4337.24
Regression8.9315.2995.982.575.78
4D-HOF5.9813.4497.271.614.40

[W2] The meaning of “feed-forward” should be clarified: does it refer to avoiding prolonged per-sequence fitting, or to requiring only a single network forward pass? The current description may lead readers to underestimate the iterative computation and gradient computation involved during inference.

By “feed-forward”, we intended to indicate generalizable frozen-weight inference via multi-step bridge transport, contrasting with prolonged per-sequence optimization. We add a footnote on page 2 to prevent any misunderstanding for readers.

Importantly, all our efficiency claims already account for this overhead: as shown in Table 3, processing a 200-frame sequence takes only ~1 minute with our method, compared to tens of minutes to hours for baselines that rely on per-sequence optimization.

[W3] The paper attributes smooth trajectories to the learned interaction prior and states that they are obtained “without explicit smoothness regularization.” However, Eq. (11) explicitly includes velocity and acceleration penalties, and the full model uses this guidance. Table 7 shows that removing temporal guidance increases JM from 1.61 to 1.91. Therefore, the full-model results alone cannot establish that smoothness comes entirely from the learned prior.

We have revised “without explicit smoothness regularization” to distinguish these two contributions precisely (revised PDF lines 415-416): our learned prior, working alongside test-time guidance, substantially improves temporal coherence and smoothness.

The ablations show that guidance alone produces only a modest improvement in JM, from 3.95 for VFM-only to 3.71 for VFM + guidance. In comparison, 4D-HOF without temporal guidance achieves a JM of 1.91, and adding temporal guidance further reduces it to 1.61. These results support a substantial contribution from the learned interaction prior, together with an additional benefit from temporal guidance.

VariantCD_hIVCAJMMPJPE
VFM-only46.5811.0678.073.954.03
VFM+guidance41.7511.8981.533.714.03
w/o guidance_temporal6.3514.5397.291.914.74
4D-HOF5.9813.4497.271.614.40

[W4] Similarly, the claim of improved physical plausibility should reflect the trade-offs across metrics. In Table 4, “VFM only” has an IV of 11.06, whereas the full model has an IV of 13.44. Although the full model substantially improves CDh and CA, it does not improve all physical metrics relative to the initialization.

The lower IV of the VFM-only variant should be interpreted together with its substantially lower contact accuracy. IV measures interpenetration, but does not penalize missing contact: a hand separated from an object can have zero interpenetration while failing to reconstruct the interaction. Consequently, lower IV alone does not establish greater physical plausibility.
VFM-only achieves an IV of 11.06, but its CA is only 78.07 and its CD_h is 46.58. Our full model increases CA to 97.27 and reduces CD_h to 5.98, with an IV of 13.44. These results show substantially improved contact recovery and geometric accuracy, alongside a tiny trade-off in interpenetration. Our claim concerns the quality of the reconstructed interaction assessed jointly across these metrics. The IV comparison alone does not negate the substantial improvements in overall physical plausibility.

AuthorsOpenReview ↗

Rebuttal 2/2

[Q1] What would the results be if the model started from Gaussian noise while using the coarse initialization as a condition? How much of the improvement comes from the structured source distribution itself?

Substituting Gaussian noise for the structured transport source while keeping the coarse initialization as conditioning results in a notable performance decline. This degradation stems from the compounded difficulty of recovering pose parameters from scratch while simultaneously modeling fine-grained hand–object interactions. In contrast, using the coarse estimate directly as the transport source establishes a structured kinematic prior early on, freeing the learned dynamics to concentrate entirely on interaction refinement. This ablation provides more direct evidence for the benefits of a structured transport source in the bridge-matching formulation.

VariantCD_hIVCAJMMPJPE
Gaussian74.1419.5488.6418.4337.24
4D-HOF5.9813.4497.271.614.40

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.