acceptodds
Withdrawn from ICLR 2027

MedManip: Scan-Anchored Hierarchical Optimization for 4D Medical Loco-Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humanoid robots have the potential to assist with specialized medical manipulation tasks, yet collecting large-scale whole-body demonstrations in this domain requires skilled operators, specialized equipment, and substantial human effort. Video generation models provide a scalable alternative, but their outputs often exhibit unstable object motion, inaccurate medical-object geometry, and implausible physical contact, which are unacceptable in medical settings. We present MedManip, a pipeline that leverages scanned medical objects as geometric anchors to convert generated interaction videos into physically grounded 4D human-object trajectories. Specifically, MedManip progressively propagates the accurate object geometry through three stages: scan-anchored object trajectory estimation, object-guided whole-body motion refinement, and geometry-aware contact optimization. We first recover a temporally stable object trajectory from the generated video, then use the recovered object motion to correct whole-body reachability and human-object alignment, and finally refine fine-grained contact while keeping the object trajectory fixed. The resulting trajectories are retargeted to a humanoid robot and converted into executable whole-body motions through low-level control. The recovered trajectories are subsequently retargeted to a humanoid robot and executed through physics-based control. Experiments on generated sequences across over 80 objects show that MedManip improves the valid demonstration rate from around 10% to over 30% and achieves higher physics-based execution success than existing generation pipelines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-Object Interaction Reconstruction, Humanoid Loco-Manipulation, Whole-Body Control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VmicOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MedManip, a pipeline for turning generated interaction videos into 4D human–object trajectories for humanoid learning, with a focus on medical objects. It uses scanned object geometry to guide reconstruction and refines the motion in three steps: correcting the object trajectory, adjusting the human body to reach and follow the object, and improving hand–object contact. The main contribution is this staged refinement process, supported by temporal depth and motion cues. The recovered trajectories are then retargeted to a humanoid for policy training. The authors evaluate the pipeline on more than 80 household and medical objects through reconstruction comparisons, ablations, and simulated G1 execution. They report improved contact quality and reconstruction usability, along with a 63.6% episode-level tracking success rate in simulation.

Critical strengths

The pipeline addresses a useful practical problem. Generated interaction videos often contain geometry and contact errors that make them difficult to use for robot learning. Using known object geometry to guide object motion, body alignment, and hand contact gives the refinement process a clear structure. This is a sensible approach to improving the usability of synthetic demonstrations.

The reconstruction results show meaningful gains. The reported screening pass rate increases from 17.2% with GRAIL to 32.6%, alongside lower contact distance and penetration. The ablations also support the usefulness of the depth, motion, and contact constraints. These results suggest that the refinements have practical value, even though they do not yet isolate the benefit of the proposed optimization order.

Critical weaknesses

The methodological contribution beyond GRAIL (Nvidia paper) is limited, and the central design choice is not directly tested. GRAIL already uses known object assets and scene geometry to guide video generation, interaction reconstruction, and humanoid control. MedManip mainly adds temporal depth and motion cues and changes the refinement into an object-to-body-to-contact sequence. However, the ablations remove individual loss terms without comparing this sequence against joint or alternating optimization using the same constraints and initialization. The results therefore do not establish whether the proposed hierarchy is responsible for the gains. Since this is the main methodological contribution, that missing comparison makes it difficult to judge the significance of the advance.

The medical framing is not well supported by the tasks or evaluation. The representative tasks mostly involve ordinary object handling, with an oral scanner included alongside household objects. The paper does not demonstrate a medical-specific manipulation requirement that changes the method or evaluation. Moreover, the reported 63.6% tracking success rate uses a mean object-position error threshold of 20 cm, and the oral-scanner task has an average error of 23.12 cm. These results support some ability to track object motion, but provide little evidence for the specialized medical manipulation emphasized in the motivation. This weakens the application contribution that might otherwise justify the limited methodological novelty.

Questions for authors

Have you compared the proposed object-to-body-to-contact sequence with joint or alternating optimization using the same initialization, loss terms, and a comparable computational budget? This would help establish whether the gains come from the hierarchy itself or mainly from the added depth and motion cues. I would also like to understand how fixing the object trajectory affects cases where its initial refinement remains inaccurate.

Is the intended application general handling of medical tools, or performing a specialized medical operation? What constitutes successful completion of the oral-scanner task beyond following its reference trajectory? Please explain how the reported position errors and the 20 cm tracking threshold relate to that task. Task-specific results and representative execution videos would help assess whether the demonstrated capability supports the paper’s medical motivation.

Any video results?

Reviewer_43pjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MedManip studies how to convert generated interaction videos into executable 4D human-object trajectories when video-derived depth, object motion, and contact are unreliable. MedManip sequentially optimizes object motion, whole-body alignment, and hand--object contact using scanned geometry, temporally consistent depth, Lift4D motion cues, and surface-contact objectives. Experiments on 80+ objects and 2 scene configurations show better physical executability and reconstruction usability than baselines. However, validation is limited to simulation.

Critical strengths

MedManip presents an object-first refinement strategy for geometric interactions, using scanned surfaces, temporal object-motion cues, and explicit contact constraints to improve generated 4D HOI trajectories. The evaluation results show gains over GRAIL in contact, penetration, object deviation, and automatic usability.

Critical weaknesses

The method primarily replaces GRAIL's joint human-object refinement with an object-first staged hierarchy and adds temporal and contact constraints. However, the ablations remove only these auxiliary losses and never isolate the hierarchy or object-freezing mechanism from a joint-optimization baseline using identical constraints. Relative to GRAIL, MedManip's technical advance is limited, and the effectiveness of its central design has not been fully demonstrated.

Questions for authors
  1. Can you compare the proposed hierarchy and object-freezing mechanism against GRAIL-style joint optimization using identical generated videos?

  2. How was the evaluation designed to exclude selection bias and support scalability?

Reviewer_rfEJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a pipeline for converting human-object interaction videos into 4D trajectories for training humanoid loco-manipulation policies. For each sequence, a human, camera, and synthetic asset are placed into a Blender environment. The rendered image is used for image-to-video generation, and the resulting video is used to recover 4D human-object interaction trajectories. The paper largely follows GRAIL's pipeline to recover loco-manipulation trajectories (e.g. Genmo to recover human trajectories, FoundationPose with known mesh to recover object trajectories, then various scene-specific fitting stages such as 2d projection consistency, aligning trajectories to Lift4D, pulling the palm towards the object, etc.) Similar to GRAIL, the trajectories are retargeted to Unitree G1 and used to train RL policies.

Critical strengths

The engineering pipeline of the paper is reasonable.

Critical weaknesses

To my knowledge, this paper is a standard 4D human-object-interaction reconstruction pipeline largely derived from GRAIL, with limited novel techniques or insights. The main engineering contributions on top of GRAIL are swapping various segmentation and depth models compared to GRAIL, incorporating Lift4D, enforcing the relative position between the object and the palm, and detecting contact via distance from object masks to hand keypoints. Many of these heuristics break in general scenarios (e.g. real-world scenes w/o GT depth, and hands coming in and out of contact with objects in dense environments). The ablations in Table 3 remove entire loss terms rather than replacing them with GRAIL's version - the depth ablation removes depth loss entirely rather than replacing it with GRAIL's version. The key contributions of the paper (palm guidance, proximity contact detector, hierarchical ordering) are never ablated.

The contribution of the pipeline is therefore rather incremental and therefore it is therefore important to assess this paper from the quality of results and the magnitude of the dataset contribution that this method could plausibly enable. Here, I see no evidence that this paper enables collecting a dataset with anything more than a single simple lifting motion (in contrast to GRAIL which provides >20k sequences with >1k objects and real robot results).

Critically, there is no evidence of motion diversity in the dataset - every qualitative example (Fig. 1-4, 6) shows a single-handed motion by a stationary person with no locomotion. No example shows loco-manipulation, nontrivial manipulation, or tool use.

The Intermimic SR of the full method in Table 3 is only 2.99%, implying <3% frame-level tracking success, whereas GRAIL's Table 1 has 88.9% SR under their settings.

The hierarchical optimization in the title is never ablated.

The video generation pipeline lacks details, e.g. which video generative model was used, prompts for the generative model, number of generated videos in the dataset, etc.

The paper claims to focus on medical objects, yet only one out of five objects in Table 2 is medical (apple, avocado, broccoli, milk are clearly not medical).

Questions for authors

As stated in weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.