acceptodds
Withdrawn from ICLR 2027

Art2Sim: Learning Physically Executable Full-Body Humanoid Interaction with Articulated Objects from Video Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video priors provide a scalable source of whole-body human-object interactions, yet visually coherent trajectories do not guarantee successful physical execution. This gap is especially acute for articulated objects. To open a door or pull a drawer, a humanoid must sustain contact with the correct moving link and coordinate whole-body motion to drive the object's passive joint. To bridge this gap, we introduce Art2Sim, a video-to-physics framework that transforms a monocular interaction video paired with an articulated asset into physically feasible full-body manipulation. Our framework consists of two stages: (1) Contact-Aware Articulated Reconstruction builds a contact-consistent kinematic reference that aligns the hand with the moving link; (2) Articulation-Guided Physics Refinement learns a residual policy on top of a pretrained whole-body tracker, using contact, joint-progress, and mechanical-work rewards to convert the reconstructed 4D HOI reference into feasible interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Object Interaction, Human Motion Synthesis, Physic Simulation, 4D Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gBNnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Art2Sim, a framework that turns a monocular interaction video and a matching articulated object asset into a physically executable humanoid interaction. It first reconstructs the human and object motion, using contact cues to align the hand with the moving object part. It then learns a residual policy on top of a pretrained whole-body tracker, with contact and joint-progress rewards to help the humanoid actually move the object through contact. The main contribution is adapting this reconstruction-and-control pipeline to articulated objects such as doors and drawers. The authors evaluate reconstruction and physical execution using real videos, generated videos, and captured interaction sequences. They report improvements over the compared baselines, with ablations showing that the progress reward is particularly important for successful manipulation.

Critical strengths
  1. Following a reference motion does not necessarily make a humanoid open a door or pull a drawer. Art2Sim explicitly connects hand contact with movement of the object’s passive joint. This focus makes the work useful for understanding what is needed to turn reconstructed motion into physically executable behavior.

  2. The physics comparisons give each method the same reference, which helps separate improvements in control from improvements in reconstruction. The ablations are also informative: adding the progress reward raises success on the captured ARCTIC sequences from 27.5% to 62.5%. This is a substantial gain and supports the practical value of explicitly rewarding joint actuation, even if the novelty of the overall approach remains a concern.

Critical weaknesses
  1. The methodological advance over closely related work appears limited. HDMI already studies video-driven whole-body interaction with articulated objects, while ResMimic uses a pretrained motion tracker with a learned residual policy for object interaction. Art2Sim adds contact-aware reconstruction and articulation-specific rewards, but it is not clear that these changes amount to a substantial advance beyond adapting existing techniques to this setting. The reconstruction and control stages also run sequentially, with no feedback from physical execution to improve the reconstructed reference. This makes the claimed coupling less substantial than the framing suggests.

  2. The experiments show that the progress reward helps, but do not establish why its particular design is needed. The strongest ablation gain comes from adding this reward, which raises success from 27.5% to 62.5%. However, the comparison is against removing the term entirely, rather than using simpler alternatives such as rewarding joint displacement or reduction in target joint error. It therefore remains unclear whether the improvement comes from the proposed phase-aware formulation or simply from providing a more direct task reward. Since this is the clearest methodological addition, that distinction matters to my assessment of the paper’s novelty.

Questions for authors
  1. What do you consider the main technical advance over HDMI and ResMimic, beyond adapting reconstruction and residual control to the articulated-object setting? In particular, does the connection between reconstruction and control provide a mechanism beyond passing a fixed, contact-aligned reference to the policy? This would help me understand where the method’s distinctive contribution lies.

  2. Have you compared the proposed phase-aware progress reward with simpler alternatives, such as rewarding joint displacement toward the target or reduction in joint tracking error, under the same training setup? The current ablation shows that adding the term helps, but does not separate the benefit of its specific formulation from that of adding a direct task reward. Such a comparison would materially affect my assessment of the central contribution.

  3. No video supplementaries.

Reviewer_3kDwOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Art2Sim turns monocular video and a matched articulated CAD asset into simulated interaction through contact-aware reconstruction and a case-specific residual controller over a frozen humanoid tracker.

Critical strengths
  1. The two-stage design addresses contact-driven articulation coherently.
  2. Shared frozen references isolate controller comparisons, and progress reward improves final-budget performance.
  3. Diverse evaluations and candid appendix disclosures clarify the method's scope.
Critical weaknesses
  1. The scalability motivation remains under-supported because no cost comparison establishes an advantage over motion capture, while costly per-case reinforcement learning still yields limited complete-interaction success and substantial tracking errors.
  2. Requiring a matched articulated CAD asset with known geometry and kinematics substantially restricts applicability to unconstrained videos, without evaluating robustness to mismatched or unavailable assets.
  3. The static results are insufficient to assess motion continuity and sustained contact consistency, and the supplementary video referenced in the paper was not included among the materials reviewed.
Questions for authors
  1. Reconcile contact weighting and the D3D-specific penalty in Section A.2 with Equations 11 and 12, and Equation 9's future-reference window with Table 7's scalar.
  2. Final-budget ablations do not establish learning speed, and Section D.1 acknowledges slow training. Add learning curves and wall-clock costs or narrow the claim.
Reviewer_77mCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Art2Sim converts a monocular video plus a matched articulated CAD asset into a physically executed full-body humanoid interaction. It has two stages: (1) Contact-aware articulated reconstruction. (2) Physics refiner. A residual PPO policy sits on top of a frozen PHC-X tracker. The evaluation covers reconstruction on D3D-HOI and on generated videos, end-to-end physics on 45 D3D-HOI cases, physics-only on 40 ARCTIC sequences, and a cumulative ablation.

Critical strengths
  1. The progress reward is effective. Tracking rewards alone do not make the policy move the object’s joint. The proposed progress reward addresses this: in the ablation (Table 4), success rises from 11/40 to 25/40, while contact matching alone gives no gain.

  2. The physics comparison is fair. All methods receive the same reference input, so the gains come from the refiner rather than a better reference.

Critical weaknesses
  1. The paper does not directly test whether contact-aware reconstruction improves physical execution. Running the same physics refiner with references from “w/o HOI Align” or baseline reconstructions would help establish the benefit of coupling the two stages.

  2. The tracking rewards are described as exp(−λE), but the error functions and weights are not specified. These details are necessary for reproducibility. The 153-dimensional action space in Eq. 8 should also be explained when first introduced, rather than only in the appendix.

  3. It is unclear how the SAM 2 anchor frame and initial prompt are selected, and whether manual input is required. The contact-labeling VLM and its prompt are also unspecified. Since these labels guide both reconstruction and policy learning, their accuracy or the method’s sensitivity to labeling errors should be evaluated.

  4. The supplementary video was unavailable to me during review. Although quantitative execution results are provided, static figures alone make it difficult to assess motion quality and contact stability.

  5. The figure formatting needs improvement. Figures 1 and 2 have insufficient spacing above their captions, and Figure 3 is too small to clearly assess the hand–object contact.

Questions for authors

Could you please clarify the points raised in the weaknesses section and address the questions listed there?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.