- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses 4D human–object–scene interaction synthesis from one image and a text instruction. Given an indoor image and a sentence such as “pull the chair, sit, pick up the cup”, a video model (MiniMax H3) generates the video, which is then used only as a rough guide for building a clean 3D animation. SAM 3D lifts the frames to object meshes and a SMPL-X body, and an occlusion heuristic guesses where hand or pelvis touches an object. They employ GPT-6 Astra as a “Director” that looks at the photo and the lifted scene, moves wrongly placed objects to match the photo, marks video frames in which an object visibly drifts (e.g. a cup rotating in the hand) as unreliable, and decides which touches are real and over which seconds they hold. Then an optimisation pushes body and objects toward the photo and the trusted frames, holds them in non-sliding contact during active intervals, keeps them out of each other and smooths them in time, with the Director revising its decisions between rounds. They evaluate 12 OMOMO actions against two text-to-motion generators and 24 self-generated videos against two video-reconstruction methods, plus a small user study and one ablation swapping the Director’s timing graph for geometric rules. Finally, a frozen InterMimic controller tracks the output in a physics simulator and the result is re-rendered as video, both as qualitative demos.
Critical strengths
-
The core idea is well motivated. The paper separates which frames to trust from when each relation’s constraints apply, so a grasp stays constrained through drifting frames instead of reproducing them.
-
The occlusion-aware contact estimator (Sec. 3.3, Eqs. 7–8) is a reusable component independent of the MLLM, which turns hand–object occlusion, normally a failure case, into contact correspondences via depth-peeled layers and self-occlusion correction.
-
The output is a structured, simulation-ready representation (explicit meshes, timestamped trajectories, typed contact events with intervals) rather than a video or a bare pose sequence, and a frozen InterMimic controller can track it in physics (Fig. 7).
Critical weaknesses
-
Nothing is compared against real captured motion. All metrics either measure how well the output matches the generated video it was fitted to, or report contact and penetration, which the optimiser enforces directly; BEHAVE and InterCap offer RGB video with ground-truth human and object poses and were not used.
-
The most similar methods are missing. ZeroHSI and HOI-PAGE also generate video and fit 4D motion with moving objects, and the recovered scene and meshes are exactly their inputs; GenHSI and FantasyHSI cover at least the static-scene cases (sitting, leaning). None is compared, although the paper itself calls ZeroHSI and GenHSI “closest”.
-
The comparison protocols are not described. Which image was used per OMOMO action, what inputs CHOIS, HOIDiNi, CRISP and RePHO received, and what the contact threshold is? A contact rate of 0.9945 means the hand touches the object in almost every frame, including reaching and releasing.
-
The main contribution, the Director, is the closed GPT-6 Astra used with unpublished prompts and code, no variance across runs, no failure analysis, and no test with another MLLM or video generator.
-
The single Director ablation (Table 4) is weak. The rule baseline finds contact in only 19.8% of annotated contact frames with unstated thresholds, so the large jump in contact rate may just reflect an untuned baseline, and the Director is slightly worse on both IoUs and foot skating. Layout correction, frame weighting and the revision loop are never ablated.
-
Units are never defined. Sec. 3.2 claims metric scale and Eq. 8 uses 0.1 m, yet the tables report “nominal scene units”. If these are metres, the Table 3(b) Pene.Mean drop is 0.15$\rightarrow$0.02 mm and the 7× claim is sub-millimetre.
-
The evaluation is small and under-reported. 24 videos, 12 actions, a 10-item user study with “30+” participants and unnamed compared methods. The same method also scores object-box IoU 0.479 in Table 3(a) but 0.874 in Table 4; the only explanation is that Table 4 is “pre-smoothing”, yet no smoothing step is described anywhere.
-
Corrections from the director are said to be penalised for moving away from the initial recovery (L215), but never how strongly, and the Fig. 4 examples (a chair turned upright, a pan moved from under a table onto it) are so large that any real penalty would have stopped them. Sec. 3.2 also measures deviation from the original recovery, so after the chair is corrected the optimiser would try to pull it back.
-
Two components are validated only qualitatively. The contact estimator is compared with InteractVLM solely by pictures (Fig. 8), with no contact accuracy or downstream ablation, and the simulation and re-rendering demos (Fig. 7) cover a few sequences with no tracking error or success rate.
-
The hand grasps in Figs. 1, 6 and 7 look too good for a method that uses no 3D ground-truth data for supervision. Fingers close neatly around cups and handles, but finger poses come only from SAM 3D Body on generated video, where fingers are mostly hidden, and no optimisation term moves fingers. Such grasps without hand–object fitting are hard to believe, and the “hand configurations” in Sec. 4.4 suggest an extra undescribed step; the authors should explain how these hands were obtained.
Questions for authors
-
Can you run recovery, Director and optimisation on BEHAVE or InterCap RGB videos and report body and object pose errors against the ground truth?
-
ZeroHSI and HOI-PAGE can run on your recovered scene and meshes; please compare, and compare GenHSI and FantasyHSI on the static-scene sequences or explain their omission.
-
For Table 1: which input image is used per OMOMO action, what is the contact threshold, what inputs did CHOIS and HOIDiNi receive, and why is contact present in 99.45% of frames?
-
What happens when each Director function (layout correction, frame weighting, revision) is switched off separately; how closely the Director’s event start and end times match the human annotations; how much results change across repeated Director runs; which contact-distance and velocity thresholds the rule baseline uses; and whether the rule baseline got the same frame weights α_t and the same chance to revise its graph.
-
Which unit does each table use, does the 0.9715 wrist proximity use the Table 4 protocol, are the main tables post-smoothing, and will prompts, harness code and videos be released?
-
What are the form and weight of the deviation penalty, how large (m, deg) were the Fig. 4 corrections, how often are proposals rejected, and is the reference in \ell_{V,t} the raw or the corrected layout?
-
Where do the finger poses in Figs. 1, 6 and 7 come from, are they optimised, taken from a template or hand-edited, and are these examples representative?