acceptodds
Withdrawn from ICLR 2027

InterGuide: Geometric Guidance for Video-based Loco-Manipulation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling imitable data is critical for learning general-purpose humanoid loco-manipulation, and in-the-wild human–object interaction (HOI) videos offer an economical and virtually unbounded source. However, motion reconstructed from such videos suffers from depth ambiguity, occlusion, temporal jitter, and often lacks detailed hand motion, making the resulting reconstructions difficult to imitate reliably via reinforcement learning (RL). We present InterGuide, a framework that leverages geometry-derived interaction guidance to refine HOI reconstruction, and formulate hand-specific rewards to facilitate imitation learning of fine-grained loco-manipulation activities. This guidance takes two complementary forms: pre-sampled coarse gripper poses produced by DenseGripper for grasp interactions, and sparse surface contact anchors for non-grasp interactions (e.g., pushing, supporting). It is integrated across three stages. First, a template-free reconstruction pipeline associates these geometric hypotheses with multi-frame video evidence to refine hand–object contact, depth ordering, and relative scale. Second, interaction-preserving retargeting maps heterogeneous human shapes onto a unified humanoid, refining the palm with the selected gripper pose while aligning gravity and ground support. Third, phase-conditioned grasp rewards use the coarse gripper poses to guide approach, acquisition, and release, while relaxing per-finger tracking to permit contact-adaptive finger motion; non-grasp interactions instead follow the refined contact anchors. Across diverse sequences spanning varied object categories, InterGuide reduces reconstructed contact error and delivers substantial gains in hand-motion imitation success rate and grasp fidelity over competitive baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Object Interaction Reconstruction, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qe9pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InterGuide learns physics-based humanoid loco-manipulation from imperfect monocular human–object interaction videos. It constructs coarse grasp hypotheses and accessible surface anchors from object geometry, then uses them for contact refinement, interaction-preserving retargeting, and phase-dependent hand rewards and termination. The pipeline requires annotated contact intervals, with optional grasp-center corrections and finger-side labels. Experiments include six non-grasp BEHAVE sequences and 25 self-captured grasping sequences. InterGuide improves contact-localization and geometric grasp-quality metrics over InterMimic and a GRAIL-inspired reward variant. Ablations examine orientation guidance and geometry-based termination, with additional comparisons of single- and multi-sequence training. All control evaluations are in simulation.

Critical strengths

The reported grasp-quality improvements are substantial across the four evaluated grasping categories.

Critical weaknesses

The methodological contribution is not sufficiently established. The complete pipeline improves the reported results, but the experiments do not isolate the benefit of the proposed geometry over simpler changes to the imitation framework, particularly relaxing unreliable finger tracking.

The evaluation does not yet establish reliable functional interaction from imperfect video references. Improved geometric scores on the selected sequences are promising, but their relationship to repeatable, effective grasping remains insufficiently verified.

Questions for authors

What accounts for the improvement over InterMimic? Beyond the shared architecture and pretrained weights, do all methods use the same reference preprocessing, reset strategy, hand joint limits, and simulation parameters? Please specify the relaxed tracking reward. The most informative additional control would be InterMimic with the same finger-tracking relaxation and initialization. Please also describe the source and scope of the category-level “clean RL replay data.”

How are evaluation rollouts executed and selected? Appendix B.1 reports the best retained rollout per sequence. How many rollouts and checkpoints were considered? Are geometry-based termination rules active during testing, and are execution termination conditions identical across methods? Please distinguish these from the success definition.

How were the sequences selected? The appendix excludes harder BEHAVE sequences because of additional contact segments, scale errors, and initial collisions. Were the evaluation sets fixed before running the methods? How many distinct object instances, participants, and environments are represented in the self-captured set, and how was the six-sequence chair ablation subset chosen?

How independent are the reconstruction targets, and how much manual intervention is needed? How often were inferred grasp centers corrected, by how much, and at what annotation cost? Were evaluation points annotated independently of the automatic suggestions? Table 1 measures contact localization rather than object shape, scale, or pose accuracy. Please clarify its reconstruction claims, why Trashbin is absent, and which categories and valid segments contribute to “All.”

What evidence supports the individual geometric components? Sparse surface anchors are evaluated through downstream non-grasp imitation, but their reconstruction benefit is not measured directly. Ordinal depth and scale adaptation are also not isolated. Please distinguish demonstrated benefits from plausible design motivations. Interaction-preserving retargeting, listed as a main contribution, would benefit from a small quantitative ablation. Finally, which training terms still use the reconstructed finger articulation after tracking relaxation? Is accurate finger reconstruction necessary, or could a mean-pose reference suffice?

Do the grasp-score gains translate into better execution? Contact preservation does not identify contact with the manipulated object. A target-object-specific contact or functional manipulation measure would therefore be useful. Please discuss the increased chair body/object errors and the table completion-rate decrease. Since errors use executed frames, could unequal temporal coverage contribute to the 1.05 cm object error for Chair InterGuideB? When alternative grasps are permitted, is the objective faithful hand-motion imitation or synthesis of a feasible substitute?

How should the baselines and ablations be interpreted? RGrailHand changes the reward aggregation and geometric measurements relative to GRAIL, so it should remain clearly identified as an adapted baseline. Its proximity reward barely changes GProxi on chairs and tables; could you explain this category-dependent behavior? In the w/o ROrient ablation, were palm-direction termination checks retained, and how often did they trigger?

Please clarify the contribution relative to RePHO (physical refinement), OmniRetarget (interaction-preserving retargeting), PHOSA (depth/scale reasoning), and HDMI, DeVI, VideoManip, and HOWTransfer (video-to-robot transfer). GPD, D-Grasp, Omnigrasp, ManipTrans, and DexMachina also provide context for grasp generation and dexterous control. Please distinguish their task assumptions; comparisons with every method are not expected. Finally, separate the demonstrated per-sequence results from the broader scaling motivation.

Reviewer_4SioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a framework to transfer HOI reconstractions to trajectory for loco-manipulation. The core idea of the paper is to introduce physical-plausible constraints, primarily contact, into reconstruction and retargeting. The paper achieves high success rate on Imitation task.

Critical strengths
  1. The paper involves multiple stages and a variety of constraints and loss terms, it presents the pipeline, design principles, and implementation details clearly overall. I find the paper easy to follow.
  2. The paper evaluates the proposed framework relatively thoroughly, from accuracy to success rate.
Critical weaknesses
  1. No real-robot evidence and no closed-loop hardware transfer. The paper normalizes scale and maps human bodies of different sizes onto a unified body shape, and the proposed strategy is shown to be effective under this single simulated embodiment. This normalization, however, shrinks the action space. Humans of different body sizes use different motions during manipulation, and robots of different sizes likewise admit different optimal trajectories at planning time. I am concerned that this step reduces the diversity of the resulting data. The paper would more convincingly establish the method if the same strategy were shown to remain effective across multiple embodiments, even if only in simulation.
  2. The paper introduces a large number of constraints, loss terms, and reward functions, but it does not thoroughly ablate them. The reported ablations remove orientation rewards and the geometry-derived termination as two groups on six chair sequences, and otherwise only compare reconstruction with and without contact refinement. I understand that the main text has limited space. Showing the practical effect of each constraint in the supplementary material would be considerably more useful to the community.
Questions for authors
  1. Does the proposed method remain effective when the initial mesh is inaccurate? which is common in real data collection
  2. Do the results change when the input is a first-person video (EGO-view), or does the method require a third-view that captures the full body?
  3. Could the authors provide a pseudocode? The paper currently describes too many stages and constraint terms, and the overview figure is too crowded to follow.
Reviewer_5siFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InterGuide learns simulated humanoid loco-manipulation from monocular human–object interaction videos. It introduces DenseGripper grasp hypotheses and accessible surface anchors, uses them to refine reconstruction and preserve contacts during retargeting, and develops phase-conditioned hand rewards and termination rules. The central contribution is sharing geometric interaction guidance across these stages while allowing flexible finger motion.

Critical strengths
  1. Relevant and well-motivated problem. Recovering usable hand–object interactions from imperfect monocular reconstructions is important for both computer vision and embodied learning. The distinction between completing a motion and establishing the intended grasp is particularly valuable.

  2. Coherent technical design. Video-conditioned hypothesis selection, preservation of object-local contacts, and geometry-guided hand control form a sensible pipeline. Separating grasp and non-grasp interactions avoids imposing inappropriate opposition constraints on supporting or pushing contacts.

  3. Encouraging empirical evidence. The reconstruction refinement improves contact localization, and the proposed method achieves better grasp metrics across the evaluated categories. Shared policy architectures, pretrained initialization, and single-sequence training budgets strengthen the controlled comparisons.

  4. Substantial implementation detail. The appendices document geometric construction, reward schedules, target assignment, annotations, and failure cases, providing useful support for reproducibility.

Critical weaknesses
  1. The evaluation does not sufficiently establish physical grasp quality. Grasp success closely follows the opposition, proximity, and orientation conditions optimized by the proposed method. Although evaluation targets are shared across methods, this primarily demonstrates satisfaction of the chosen geometric criteria. The paper also acknowledges that contact preservation cannot identify whether contact involves the manipulated object, while tracking metrics measure root translation rather than articulation or rotation. Independent evidence of stable, task-effective grasping is therefore limited.

  2. Reported success is based on selected rollouts rather than policy reliability. The paper reports the best retained rollout per sequence, without repeated-trial statistics or uncertainty estimates. Errors are averaged over executed frames, potentially favoring trajectories that terminate before difficult portions. This protocol can support successful trajectory generation, but it does not establish expected execution performance.

  3. The contribution of manual supervision is insufficiently quantified. Contact intervals are annotated, and inferred grasp centers can be corrected before retargeting and imitation. Finger-side annotations additionally activate important orientation guidance. These inputs are disclosed, but their frequency, annotation cost, and effect on performance are not measured. Consequently, the extent to which improvements arise from automatic geometric reasoning versus manual correction remains unclear.

  4. Novelty and component attribution need stronger evidence. CARI4D already provides contact-aware reconstruction [1], InterMimic addresses imitation and retargeting from imperfect interactions [2], Lightning Grasp provides procedural grasp synthesis [3], and GRAIL uses contact, opposition, and proximity rewards [4]. InterGuide’s distinctive contribution appears to be their integration through video-selected geometric hypotheses, rather than these ingredients individually. arxiv.org However, the experiments do not isolate DenseGripper against simpler geometric proposals or separately establish the value of retargeting, scale correction, and depth-order refinement. RGrailHand is also an adapted reward baseline, so its results should not be interpreted as a comparison against the complete GRAIL system.

[1] Xie et al. CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction, 2025.

[2] Xu et al. InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions, CVPR 2025.

[3] Yin and Abbeel. Lightning Grasp: High Performance Procedural Grasp Synthesis with Contact Fields, 2025.

[4] Xie et al. GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors, 2026.

Questions for authors
  1. Report average success over multiple training seeds and evaluation rollouts, alongside best-rollout results. Specify the number of candidate rollouts and account explicitly for early failures.

  2. Add object-specific contact measurements, slip or retention tests where applicable, penetration measures, and object orientation error. For reconstruction, supplement grasp-point distance with broader pose accuracy and image-consistency measures.

  3. Compare automatic grasp selection against manually corrected selection, and evaluate performance without finger-side labels. Report annotation time and correction frequency.

  4. Extend ablations across object categories, including simpler grasp proposals, contact-preserving retargeting, and individual reconstruction refinements. Test sensitivity to mesh, scale, and contact-timing errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.