acceptodds
Withdrawn from ICLR 2027

EmboDirector: Directing 4D Human–Scene Interactions from a Single Image

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Full-body human–scene interaction data with consistent geometry, motion, and contact is scarce and costly to capture. Video models acquire rich interaction priors through large-scale pretraining, yet generated interactions can exhibit object-orientation drift that direct reconstruction would reproduce. We present EmboDirector, which reconstructs structured 4D interactions from videos generated using a single indoor image and an interaction description. A multimodal large language model (MLLM) Director corrects the initial recovered layout, assigns frame reliability to reduce the influence of video drift, and organizes estimated contact candidates into a spatiotemporal interaction graph. The graph specifies interaction relationships and their active intervals, guiding joint optimization of human and object trajectories alongside reliable visual evidence. The input image anchors layout corrections throughout. Comparisons with HOI generation and video-based reconstruction baselines show improved contact rate, mean penetration, and object alignment; an ablation examines graph-conditioned optimization. The reconstructed scenes support physics simulation and geometry-guided video synthesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human–scene–object interaction, Video generative models, MLLM agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DxahOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses 4D human–object–scene interaction synthesis from one image and a text instruction. Given an indoor image and a sentence such as “pull the chair, sit, pick up the cup”, a video model (MiniMax H3) generates the video, which is then used only as a rough guide for building a clean 3D animation. SAM 3D lifts the frames to object meshes and a SMPL-X body, and an occlusion heuristic guesses where hand or pelvis touches an object. They employ GPT-6 Astra as a “Director” that looks at the photo and the lifted scene, moves wrongly placed objects to match the photo, marks video frames in which an object visibly drifts (e.g. a cup rotating in the hand) as unreliable, and decides which touches are real and over which seconds they hold. Then an optimisation pushes body and objects toward the photo and the trusted frames, holds them in non-sliding contact during active intervals, keeps them out of each other and smooths them in time, with the Director revising its decisions between rounds. They evaluate 12 OMOMO actions against two text-to-motion generators and 24 self-generated videos against two video-reconstruction methods, plus a small user study and one ablation swapping the Director’s timing graph for geometric rules. Finally, a frozen InterMimic controller tracks the output in a physics simulator and the result is re-rendered as video, both as qualitative demos.

Critical strengths
  1. The core idea is well motivated. The paper separates which frames to trust from when each relation’s constraints apply, so a grasp stays constrained through drifting frames instead of reproducing them.

  2. The occlusion-aware contact estimator (Sec. 3.3, Eqs. 7–8) is a reusable component independent of the MLLM, which turns hand–object occlusion, normally a failure case, into contact correspondences via depth-peeled layers and self-occlusion correction.

  3. The output is a structured, simulation-ready representation (explicit meshes, timestamped trajectories, typed contact events with intervals) rather than a video or a bare pose sequence, and a frozen InterMimic controller can track it in physics (Fig. 7).

Critical weaknesses
  1. Nothing is compared against real captured motion. All metrics either measure how well the output matches the generated video it was fitted to, or report contact and penetration, which the optimiser enforces directly; BEHAVE and InterCap offer RGB video with ground-truth human and object poses and were not used.

  2. The most similar methods are missing. ZeroHSI and HOI-PAGE also generate video and fit 4D motion with moving objects, and the recovered scene and meshes are exactly their inputs; GenHSI and FantasyHSI cover at least the static-scene cases (sitting, leaning). None is compared, although the paper itself calls ZeroHSI and GenHSI “closest”.

  3. The comparison protocols are not described. Which image was used per OMOMO action, what inputs CHOIS, HOIDiNi, CRISP and RePHO received, and what the contact threshold is? A contact rate of 0.9945 means the hand touches the object in almost every frame, including reaching and releasing.

  4. The main contribution, the Director, is the closed GPT-6 Astra used with unpublished prompts and code, no variance across runs, no failure analysis, and no test with another MLLM or video generator.

  5. The single Director ablation (Table 4) is weak. The rule baseline finds contact in only 19.8% of annotated contact frames with unstated thresholds, so the large jump in contact rate (0.2→0.89)(0.2 \rightarrow 0.89) may just reflect an untuned baseline, and the Director is slightly worse on both IoUs and foot skating. Layout correction, frame weighting and the revision loop are never ablated.

  6. Units are never defined. Sec. 3.2 claims metric scale and Eq. 8 uses 0.1 m, yet the tables report “nominal scene units”. If these are metres, the Table 3(b) Pene.Mean drop is 0.15$\rightarrow$0.02 mm and the 7× claim is sub-millimetre.

  7. The evaluation is small and under-reported. 24 videos, 12 actions, a 10-item user study with “30+” participants and unnamed compared methods. The same method also scores object-box IoU 0.479 in Table 3(a) but 0.874 in Table 4; the only explanation is that Table 4 is “pre-smoothing”, yet no smoothing step is described anywhere.

  8. Corrections from the director are said to be penalised for moving away from the initial recovery (L215), but never how strongly, and the Fig. 4 examples (a chair turned upright, a pan moved from under a table onto it) are so large that any real penalty would have stopped them. Sec. 3.2 also measures deviation from the original recovery, so after the chair is corrected the optimiser would try to pull it back.

  9. Two components are validated only qualitatively. The contact estimator is compared with InteractVLM solely by pictures (Fig. 8), with no contact accuracy or downstream ablation, and the simulation and re-rendering demos (Fig. 7) cover a few sequences with no tracking error or success rate.

  10. The hand grasps in Figs. 1, 6 and 7 look too good for a method that uses no 3D ground-truth data for supervision. Fingers close neatly around cups and handles, but finger poses come only from SAM 3D Body on generated video, where fingers are mostly hidden, and no optimisation term moves fingers. Such grasps without hand–object fitting are hard to believe, and the “hand configurations” in Sec. 4.4 suggest an extra undescribed step; the authors should explain how these hands were obtained.

Questions for authors
  1. Can you run recovery, Director and optimisation on BEHAVE or InterCap RGB videos and report body and object pose errors against the ground truth?

  2. ZeroHSI and HOI-PAGE can run on your recovered scene and meshes; please compare, and compare GenHSI and FantasyHSI on the static-scene sequences or explain their omission.

  3. For Table 1: which input image is used per OMOMO action, what is the contact threshold, what inputs did CHOIS and HOIDiNi receive, and why is contact present in 99.45% of frames?

  4. What happens when each Director function (layout correction, frame weighting, revision) is switched off separately; how closely the Director’s event start and end times match the human annotations; how much results change across repeated Director runs; which contact-distance and velocity thresholds the rule baseline uses; and whether the rule baseline got the same frame weights α_t and the same chance to revise its graph.

  5. Which unit does each table use, does the 0.9715 wrist proximity use the Table 4 protocol, are the main tables post-smoothing, and will prompts, harness code and videos be released?

  6. What are the form and weight of the deviation penalty, how large (m, deg) were the Fig. 4 corrections, how often are proposals rejected, and is the reference in \ell_{V,t} the raw or the corrected layout?

  7. Where do the finger poses in Figs. 1, 6 and 7 come from, are they optimised, taken from a template or hand-edited, and are these examples representative?

Reviewer_nMFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the generation of 4D human object interactions via distilation from video models. Given an input image and a text instruction they generate a video and then reconstruct the human-object interaction from it. They propose an optimization framework which uses an LLM director to estimate a spatio-temporal interaction graph, which is used in the subsequent optimization. The director predicts multiple attribute such as contact relations, frame quality, interaction time intervals. The reconstruction alternates between graph construction/reconstruction and optimization. The method improves on penetration and contact metrics and qualitative results look promising.

Critical strengths
  • The qualitative results, especially the video look very good.
  • An analysis of what attributes the LLM can extract and use for the optimization step could be quite valuable.
Critical weaknesses
  • The paper proposes a reconstruction method, but it is not clear which components are critical. The director (gpt astra) produces a lot of variables, such as contact relations, contact timings, frame quality, contact types etc. which are used in the subsequent optimization step. Which of those are important for the reconstruction is not evaluated. How some of them are used in the optimization is not even explained, e.g. what is the precise difference between a touch, grasp, support or release? A more detailed ablation study would help here. Furthermore, the method implements a loop which alternates the application of the director with the optimization, but the importance of this loop is not clear. Does it saturate at some point and what is a good amount of iterations? The same applies for the proposed occlusion aware contact estimation. Overall, i would suggest a more detailed ablation study and a clearer description of the method to improve the paper. also a lot of details are missing

  • A comparison with HOI-Page is missing and is important as their key idea is very similar since they use a VLM to predict a relation graph which influences the terms in the optimization. In addition, i would suggest to compare the reconstruction quality on datasets with GT poses, objects and contact such as BEHAVE or Intercap. This would allow a more precise evaluation and would could even aid the ablation study.

Questions for authors
  • Which models are used in the optimization pipeline?
  • How often is the loop of director and optimization step applied?
  • How does it perform on objects that are not part of the typical datasets? That is the biggest advantage of distilling from video models.
Reviewer_WuoLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EmboDirector reconstructs 4D human–object interactions from a single indoor image and a text description. It generates an interaction video with MiniMax H3, recovers initial geometry with SAM 3D models, and estimates contact candidates. An MLLM "Director" then corrects the initial layout, down-weights drifting frames, and organizes the contacts into a spatiotemporal interaction graph, which guides a joint optimization of human and object trajectories. Experiments compare against HOI generation and video-based reconstruction baselines, ablate the graph, and demonstrate simulation and video re-synthesis.

Critical strengths
  • The problem is practical and timely: video priors are used as a source of multi-stage, multi-object scene interactions. Generated videos can contain plausible frames but inconsistent object motion. Explicit contact relationships help avoid simply reproducing those errors

  • Separating frame reliability (visual evidence) from event confidence (relation assignment) is a sensible design. It lets a grasp persist through drifting frames.

  • The demos, though limited, are convincing, the end-to-end pipeline (image → 4D → simulation and re-rendering) is broad, and the limitations are stated.

Critical weaknesses
  • the results are small which only includes 12 actions, and it involves trade offs, Maximum penetration is worse than HOIDiNi’s, text–motion retrieval is worse than CHOIS’s, and the Director graph slightly increases foot skating.

  • The Director’s contributions are not fully isolated. The graph ablation retains Director-corrected layouts and other shared processing. Separate ablations are needed for layout correction, frame weighting, and graph revision. There is no test with other MLLMs. It is unclear how much of the result depends on GPT-6 Astra specifically.

Questions for authors
  1. Can you evaluate on a dataset with ground-truth 4D HOI (e.g., BEHAVE or InterCap, or rendered OMOMO/TRUMANS) using MPJPE and object-pose errors?
  2. Please add quantitative ablations removing (a) layout correction and (b) frame reliability α_t, on videos with known drift.
  3. How sensitive are the results to the MLLM? Can you report results with an open model and run-to-run variance, and will the prompts and harness be released?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.