acceptodds
Under review as a conference paper at ICLR 2027

4DHOISolver: Efficient and Accurate Human–Object Motion Reconstruction from Monocular Videos

Abstract

Internet videos capture a rich diversity of human activities, objects, and environments, offering a readily available source of interaction data for embodied intelligence. However, recovering accurate and physically plausible 4D human–object interactions (HOI) from these monocular videos remains challenging due to depth ambiguity, occlusion, and complex contact relationships. We introduce 4DHOISolver, a pipeline that combines automatic reconstruction with efficient manual correction. Our agent-based reconstruction method uses semantic interaction reasoning to guide geometric optimization, but challenging in-the-wild videos can still lead to reconstruction failures. These failures motivate an efficient and scalable manual repair method that corrects reconstructed motion through simple, sparse point annotations. We further propose a unified physical refinement method that corrects residual contact errors and interpenetration across diverse interactions without action-specific reward design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Object-Interaction, 3D Reconstruction, Human-in-the-Loop, Humanoid

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NX5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

4DHOISolver reconstructs whole-body human-object motion from monocular video in three stages. It starts with agent-assisted initialization and geometric optimization, uses sparse point-pair annotations to repair difficult sequences, and then refines the human motion in simulation. The evaluation covers InterCAP and 200 in-the-wild sequences, with a 20-case annotation study and physical-refinement experiments on 45 motions.

Critical strengths
  1. For me, the repair tool is the strongest contribution. The point-pair interface gives users a direct way to correct contact and alignment, and the 20-case study shows less hands-on annotation time, lower penetration, and better ratings from blinded evaluators. This is concrete evidence that the tool can improve a practical reconstruction workflow.
  2. I appreciate the breadth of the in-the-wild evaluation: 200 sequences across 92 object categories. It tests the workflow across varied appearances and interactions, which makes its potential usefulness more convincing than a few selected demonstrations would.
Critical weaknesses
  1. I'm less convinced by the automatic stage. On the wild set, its penetration is higher than CARI4D's (33.99 versus 20.14 mm). The large contact-distance improvement after repair is measured against the same annotations used to guide it. This leaves limited independent evidence that the automatic reconstruction improves on prior methods, which matters because it is presented as a main contribution.
  2. The contribution of physical refinement is also unclear. Object trajectories are prescribed, object gravity is disabled, and a separate PPO controller is trained for each sequence. Without a contact-reward ablation, I cannot tell what the proposed term adds beyond the underlying tracker. The results support a narrower tracking setup than a general claim about physically coupled human-object interaction.

Missing references: AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation (SIGGRAPH 2026), HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions (ECCV 2026).

Questions for authors
  1. Is there an independent pose or contact reference for evaluating the automatic stage on wild videos? Could you also clarify how the 20 repair-study cases were selected, since that affects how broadly I can interpret the workflow results?
  2. Do you have a contact-reward ablation on the same 45 motions with other settings fixed? Please also report how many motions contribute to each penetration metric after unreliable meshes are excluded. These details would determine how much support I place on the physical-refinement result.
  3. Could you give the processing time and manual effort for each stage, including per-sequence PPO training? The current hands-on timing is useful, but the full cost would affect my assessment of the claimed efficiency.
2 comments
AuthorsOpenReview ↗

Automatic Reconstruction, Contact-Reward Ablation, and Evaluation Costs

1. Automatic reconstruction and contact evaluation

We agree that our automatic result has higher penetration than CARI4D (33.99 vs. 20.14 mm). Contact and penetration should be considered together: severe contact errors, defined as a per-video median contact distance above 20 cm, occur in 24/200 videos (12%) for our method and 48/200 (24%) for CARI4D. On the 94 videos where both methods achieve contact distance at most 20 cm and penetration p95 at most 5 cm, our median contact distance improves from 66.32 to 60.24 mm, while penetration increases from 16.14 to 18.45 mm. We will report the full-set results alongside this subset analysis and state the selection criteria explicitly.

The contact annotations are not inputs to either automatic method, so they provide an independent measure of automatic contact alignment. After point-pair repair, the same annotations become optimization targets; we interpret the larger improvement at that stage as constraint satisfaction, not independent reconstruction accuracy.

2. Wild-video reference and repair-study selection

We do not have independent 3D pose or contact ground truth for the wild videos. Table 2 therefore evaluates contact and physical consistency rather than absolute reconstruction accuracy. The 20 repair-study cases were selected from internet and self-recorded videos with visible errors in the automatic results. They cover 20 different object categories and varied interactions. We will clarify that this is a targeted repair study and describe its selection procedure.

3. Contact-reward ablation and penetration sample counts

We completed the contact-reward ablation on the same 45-motion test set with all other training settings fixed, using the original evaluation protocol. Adding the reward reduces reference-tracking MPJPE from 95.25 to 94.62 mm, angular jerk from 195.60 to 183.38 rad/s³, penetration depth from 3.15 to 2.79 mm, and penetrating frames from 19.39% to 17.96%. These results support the contribution of the contact reward beyond the base tracker under prescribed object motion. Regarding the effective sample counts, some object meshes are non-watertight and do not support reliable signed-distance evaluation, so these motions were excluded from the penetration metrics as described in Appendix C.4. Both penetration metrics use 20 motions. This mesh-validity filtering applies only to penetration evaluation, not to MPJPE or angular jerk.

4. Processing time and manual effort

Our efficiency claim concerns annotation and geometric optimization. For a 273-frame clip matching the benchmark’s median automatic-input length, VLM-guided automatic optimization takes 2.26 min, including Qwen and frame preparation and 1,500 Adam steps. In the 20-case study, hands-on annotation takes 40.63 s per video, while point-guided geometric optimization takes approximately 1.82 min for a typical-length clip. Physics-based PPO is a separate, optional refinement stage. Its training costs 7.38 GPU-hours per sequence for our method versus 18.09 GPU-hours for the adapted RePHO baseline, accounting for both GPUs used in training. We will report the timing protocol and hardware in the supplement.

5. Related work

Thank you for identifying the missing references. We will add and discuss AGILE and HSImul3R in the related-work section.

Thank you for the response and the additional experiments. The InterCAP repair results and contact-reward ablation address some of my concerns.

However, the automatic result still has higher penetration than CARI4D on the wild set. The gains from the contact reward are also modest, and penetration is evaluated on only 20 of the 45 motions. Please make these limitations, the prescribed object motion, and the full processing cost clear in the paper.

For now, I’ll keep my score at 2.

Reviewer_nM33OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents 4DHOISolver, a three stage pipeline for reconstructing 4D HOI from monocular in the wild videos. It utilizes a VLM to guide the geometric optimization of the off-the-shelf foundation models. Then, it introduce a sparse annotation interface to optimize the tracking results. Finally, the physics based control policy augmented with a fine-grained contact reward to reduce interpenetration and improve contact plausibility. The results looks good after the entire pipeline.

Critical strengths

The biggest strength of the paper is that the sparse point reconstruction part looks contribute significantly to the HOI reconstruction result. This gives some interesting view for data capturing process.

Critical weaknesses

There are three major weaknesses:

  1. The clarity of human annotation requirements need to be improved. The annotation density, reported time breakdown, and the detail of supporting InterPoint model requires additional details.
  2. The tracking and simulation setup is merely learning a sequence-memorized trajectory to press its links against a moving kinematic obstacle.
  3. The claim of "agent-based" is somewhat over-claimed. It seems that the VLM is not acting like a closed loop autonomous agent that iterates over a sequences of steps. Instead, it is used in a few steps of selection and semantic prediction. Removing the agent's contact guidance also has limited gain. I wouldn't call it an "agent".
Questions for authors
  1. What are the training detail for the InterPoint model (data, method)? How it is used in the annotation pipeline?
  2. How many annotations are needed from the human? Also, the reported time is very vague in terms what exactly contributes to it.
  3. Is the tracking part only used to avoid collision?
2 comments
AuthorsOpenReview ↗

Annotation Protocol, Simulation Scope, and VLM Terminology

1. Annotation requirements and InterPoint

Each benchmark video requires an average of 3.80 distinct 3D human–object contact correspondences and 2.29 object–image correspondence tracks. The 40.63 s in Table 3 was measured without InterPoint for a fair comparison. It covers manual frame selection, object-surface point selection, human-joint selection, and 2D image tracking-point selection; subsequent processing is excluded.

InterPoint is an optional proposal tool: annotators verify or correct its contact pairs before optimization. It combines image features from LLaVA with PointNet++ object features, and is trained using human-contact classification and bidirectional point-matching losses. It was trained on 360 annotated videos. On 43 training-disjoint videos, it correctly proposed 1.44 of 4.67 annotated pairs per video on average, corresponding to 30.85% pair recall. We will add the training and evaluation protocols to the supplement.

2. Purpose and scope of simulation tracking

Collision reduction is one purpose of this stage, but not its only objective. The sequence-specific controller tracks the reconstructed human motion under simulated human dynamics, while our contact reward encourages contacts at the annotated body regions, object locations, and time intervals. The object trajectory is prescribed, and a separate controller is trained for each sequence. We will clarify that this stage refines human motion and contact under prescribed object motion; it does not show that human forces generate the object trajectory.

3. “Agent-based” terminology

We agree that the current pipeline is more accurately described as VLM-guided: the VLM makes selections and predicts semantic priors, but does not operate in a closed loop. We will update the terminology and avoid overstating the modest contact-guidance gain in Table 1. Feeding reconstruction results back to the VLM to revise its decisions is a natural next step, which we plan to develop and evaluate in future work.

Thank you for the clarifications and additional details. The annotation cost explanation clarify some of my concerns.

However, given how minimal the contact-guidance gain, the reliance on manual annotation and its costs remains non-negligible. Also, prescribing kinematic object motion without gravity, training per-sequence policies, and reporting modest contact-reward gains on a filtered subset still limit this stage to a narrow tracking refinement rather than coupled HOI physics.

I appreciate the responses, but given the remaining simulation and evaluation limitations, I will keep my score at 2.

Reviewer_hyeBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is a pipeline for 4d human-object interaction reconstruction, built on three stages. First, an automatic human-object estimation pipeline similar to CARI4D is run, swapping in newer models such as Sam3D, as well as adding a scene-guided ground support loss for objects, and a new Qwen3VL-based contact estimation stage that replaces CARI4D's contact prediction network. Second, a manual repair step is introduced where a human annotator assigns soft relationships between 3D object points and 3D human keypoints as well as 3D object points and 2D image points. Finally a physics stage is applied that freezes the object trajectory and optimizes the human motion to be more physically plausible. Here, a motion tracking policy is trained for a RePHO humanoid following the ProtoMotions3 recipe. A contact reward is applied where the humanoid is rewarded for bringing the links corresponding to annotated 3D human keypoints into contact with the corresponding frozen 3D object points along the trajectory. The automatic reconstruction stage is evaluated on InterCAP and the human annotation stage and physics stage are evaluated on a self-curated mixture of Internet and cellphone videos.

Critical strengths

A human annotation interface that attaches the object to human and image keypoints can improve performance on top of the automatic methods such as CARI4D, and is valuable for improving quality of human-object-interaction reconstruction in scenarios where the data annotation pipeline can work.

As shown in Table 3, the annotation strategy of attaching 3D object points to both 3D human keypoints and 2D image keypoints is effective.

Using an optimization stage and then a physics simulator to ensure the annotated relationships are met seems reasonable.

Critical weaknesses

In the video results and in Fig. 15, the method seems to only be demonstrated on sequences where either the object is largely static (e.g. woman sitting on bench, step-on stool, sofa), or the relative pose between the object and one of the human keypoints is mostly fixed throughout the entire video (e.g. badminton racket, playing flute or cello, chair pushing). For example, in the chair pushing video, the human is never shown putting hands onto the chair or taking hands off at the end of the motion. However, many real-world videos (such as several InterCAP sequences) have evolving human-object contact relationships throughout the video (to a larger degree than e.g. human playing flute, where hands do not significantly move relative to the instrument throughout the video) and it is unclear whether the method as a whole achieves good results on these more complex tasks (e.g. picking up or putting down an object, adjusting the grip on an object during the video, etc.).

The full method (including the human annotation interface and physics-based human pose refinement) is never evaluated on a benchmark with ground-truth: it's only evaluated on in-the-wild clips whose composition and details are not disclosed. It is therefore unclear whether the in-the-wild benchmark is simpler than InterCAP/BEHAVE videos, and whether annotation interface and physics refinement can improve results on more complex interaction sequences. All qualitative results on InterCAP in Figure 7 (and the qualitative results in the supplementary video) are of the automatic reconstruction pipeline (with and without contact guidance). Similarly Table 1 only shows automatic reconstruction result and does not show the result w/ human annotation. Table 2 is evaluated only on in-the-wild sequences, not on standard benchmarks with ground-truth. Evidence that human annotation interface and physics refinement can improve results on benchmarks with ground-truth (e.g. similar to InterCAP or BEHAVE) would improve the paper.

For dynamic objects, the physics optimization stage seems weaker than prior works (InterMimic, RePHO) which also simulate the object, so the object's motion must come from simulated contact forces (although InterMimic/RePHO's tracking may fail). Here, it seems only the human is simulated around a kinematically replayed object trajectory. Since the object might be floating or its motion might not be caused by contact and forces exerted by the human, it's not necessarily fair to claim "physically plausible HOI motion".

The differences in the automatic reconstruction pipeline are minor compared to prior work, aside from the annotation interface which might have limitations as in the first paragraph. The main difference in the automatic reconstruction pipeline compared to CARI4D seems to be swapping in newer models (e.g. SAM3D), as well as the scene-guided optimization which encourages objects to lie on the ground plane, and the contact-guided optimization part where Qwen3VL agent produces contact estimates instead of CARI4D's contact prediction network. And the contact-guided optimization does not have significant difference compared to the version without contact guidance as shown in Table 1.

As stated in the paper, Table 2 is evaluated against the human contact annotations themselves (which are the optimization targets) and not any ground-truth - it is not a measure of reconstruction accuracy.

The "45-motion test subset" in Section 5.5 is not explained. Appendix B.1 claims physical refinement uses a subset of this benchmark explained below, but the explanation does not seem to be there.

The 200-sequence wild benchmark is not explained. There is no discussion on length of the videos, amount and complexity of the motion, what the 92 objects are, interaction types, etc.

Most of the loss terms in Eq 2 are not explained.

The effectiveness of InterPoint is not evaluated.

Questions for authors

Multiple questions are raised in weaknesses above.

How does the contact annotation interface work when the object is not static and there is a clear dynamic human-object interaction in the video (e.g. the person approaches an object and makes contact during the video, the person moves a chair into place then sits down, etc.)

Describe the composition and details of the in-the-wild evaluation benchmark? How many videos in the evaluation benchmark contain objects that are fully static (e.g. the two videos of women sitting on benches in the supplementary video) or involve no contact changes throughout the video?

1 comment
AuthorsOpenReview ↗

Dynamic Contacts, InterCAP Repair Results, and Evaluation Clarifications

1. Dynamic contacts and annotation updates

Thank you for pointing out that our video examples do not clearly show contact transitions. Our method supports interactions in which human–object contact changes over time. Of the 200 benchmark sequences, 181 involve dynamic objects and 81 contain a change in contact state or location. Our interface does not assume persistent contact. When a person touches an object, changes their grip, or lets go, the annotator updates or removes the relevant point pair. During non-contact intervals, object-to-image correspondences and temporal constraints guide object alignment without imposing human–object contact.

Rows 1, 3, 4, and 6 of supplementary Fig. 15 contain contact changes. We will add video examples that clearly show contact onset and release.

2. Ground-truth evaluation of point-pair repair

We have evaluated point-pair repair on InterCAP using its 3D ground truth. Relative to our automatic result, object Chamfer distance improves from 16.57 to 15.31 cm, and combined Chamfer distance from 12.53 to 12.13 cm. Both repaired results also improve upon CARI4D’s 15.69 and 12.88 cm, respectively. We will add these results to Table 1 and include corresponding qualitative examples.

3. Scope of physical refinement

Our physical refinement dynamically simulates the human while prescribing the reconstructed object trajectory. It improves human motion and contact under that object motion; it does not establish that human contact forces generate the object trajectory. We will qualify our physical-plausibility claims accordingly. For Table 4, we adapt RePHO to use the same prescribed object trajectories during training and evaluation. This comparison therefore evaluates refinement under a shared object-motion setting.

4. Role of the automatic stage

The VLM assists with interacting-pair and reconstruction-frame selection, then provides contact-phase, body-region, object-motion, and ground-support priors for geometric refinement. Depth-based human–object alignment and robust object tracking produce the temporally coherent initialization used by point-pair repair. We will clarify these contributions relative to CARI4D.

5. Interpretation of Table 2

As stated in Sec. 5.3, contact distance measures agreement with annotated contacts, not ground-truth 3D reconstruction accuracy. For CARI4D and our automatic result, the annotations are used only for evaluation, providing an independent measure of contact alignment. After point-pair repair, they become optimization targets, so the metric measures constraint satisfaction. The evaluation measures distance from an annotated human contact point to the object surface, whereas repair constrains specific human–object point pairs. We will clarify this distinction in the table caption.

6. Additional benchmark and method details

  • The 200 clips span 92 object categories, with at most 12 clips per category. Their median duration is 9.12 s, with a median of 258 reconstructed frames. Scene annotations classify 19 objects as static and 181 as dynamic; 81 clips have contact changes, while 119 do not. Reconstructed objects have a median maximum spatial excursion of 0.517 m. Interactions span carrying or lifting (55 clips), musical performance (44), sports or exercise (36), tool use (29), other touching, holding, or demonstration activities (17), drinking or other everyday activities (14), pushing or pulling (11), standing or leaning support (7), and sitting or reclining (7); these preliminary labels may overlap. We will add the complete category counts to the supplement.

  • The 45-motion test set is a subset of the 200-video benchmark and covers 29 object categories. We will describe its selection and composition in Sec. 5.5 and Appendix B.1, and correct the incomplete cross-reference. We will also expand Eq. 2 to define all six losses, optimized variables, valid-frame masks, and term weights.

  • Finally, we evaluated InterPoint on 43 videos excluded from training. These videos contain an average of 4.67 annotated 3D contact pairs each. InterPoint correctly proposes 1.44 pairs per video on average, corresponding to 30.85% pair recall. Annotators verify or correct proposals before optimization. We will add the evaluation protocol and results to the supplement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.