- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Art2Sim, a framework that turns a monocular interaction video and a matching articulated object asset into a physically executable humanoid interaction. It first reconstructs the human and object motion, using contact cues to align the hand with the moving object part. It then learns a residual policy on top of a pretrained whole-body tracker, with contact and joint-progress rewards to help the humanoid actually move the object through contact. The main contribution is adapting this reconstruction-and-control pipeline to articulated objects such as doors and drawers. The authors evaluate reconstruction and physical execution using real videos, generated videos, and captured interaction sequences. They report improvements over the compared baselines, with ablations showing that the progress reward is particularly important for successful manipulation.
Critical strengths
-
Following a reference motion does not necessarily make a humanoid open a door or pull a drawer. Art2Sim explicitly connects hand contact with movement of the object’s passive joint. This focus makes the work useful for understanding what is needed to turn reconstructed motion into physically executable behavior.
-
The physics comparisons give each method the same reference, which helps separate improvements in control from improvements in reconstruction. The ablations are also informative: adding the progress reward raises success on the captured ARCTIC sequences from 27.5% to 62.5%. This is a substantial gain and supports the practical value of explicitly rewarding joint actuation, even if the novelty of the overall approach remains a concern.
Critical weaknesses
-
The methodological advance over closely related work appears limited. HDMI already studies video-driven whole-body interaction with articulated objects, while ResMimic uses a pretrained motion tracker with a learned residual policy for object interaction. Art2Sim adds contact-aware reconstruction and articulation-specific rewards, but it is not clear that these changes amount to a substantial advance beyond adapting existing techniques to this setting. The reconstruction and control stages also run sequentially, with no feedback from physical execution to improve the reconstructed reference. This makes the claimed coupling less substantial than the framing suggests.
-
The experiments show that the progress reward helps, but do not establish why its particular design is needed. The strongest ablation gain comes from adding this reward, which raises success from 27.5% to 62.5%. However, the comparison is against removing the term entirely, rather than using simpler alternatives such as rewarding joint displacement or reduction in target joint error. It therefore remains unclear whether the improvement comes from the proposed phase-aware formulation or simply from providing a more direct task reward. Since this is the clearest methodological addition, that distinction matters to my assessment of the paper’s novelty.
Questions for authors
-
What do you consider the main technical advance over HDMI and ResMimic, beyond adapting reconstruction and residual control to the articulated-object setting? In particular, does the connection between reconstruction and control provide a mechanism beyond passing a fixed, contact-aligned reference to the policy? This would help me understand where the method’s distinctive contribution lies.
-
Have you compared the proposed phase-aware progress reward with simpler alternatives, such as rewarding joint displacement toward the target or reduction in joint tracking error, under the same training setup? The current ablation shows that adding the term helps, but does not separate the benefit of its specific formulation from that of adding a direct task reward. Such a comparison would materially affect my assessment of the central contribution.
-
No video supplementaries.