acceptodds
Under review as a conference paper at ICLR 2027

IMAGIN-4D: Image-Guided Controllable Interaction Generation

Abstract

Generating human-object interactions (HOI) is central to character animation, robotics, AR/VR, and embodied AI. Existing methods use text, object geometry, and sparse waypoints to control actions and object trajectories, but leave grasps, approach directions, body and object poses, contacts, and relative layouts ambiguous. We use a reference image to specify the desired interaction snapshot. However, a single global image representation conflates distinct cues and uses the same visual evidence to condition every frame. We introduce IMAGIN-4D, a diffusion-based HOI generator with spatio-temporal image conditioning. For spatial conditioning, supervised interaction-state tokens encode body pose, object pose, body-object contact, and spatial relationships at the depicted frame. For temporal conditioning, frame-aware tokens query image patches per generated frame to retrieve visual cues. Role-aware conditioning balances these signals: text, waypoints, and interaction-state tokens use separate AdaLN streams, while motion tokens cross-attend to frame-aware visual tokens. Since HOI motion datasets lack paired images, we render synthetic references from FullBodyManipulation (FBM) and introduce an image-adherence metric measuring agreement with the reference snapshot. Experiments on FBM and BEHAVE show that IMAGIN-4D improves fine-grained interaction control over single-token and uniformly image-conditioned baselines while preserving waypoint-following and motion quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human object-interaction, 4d interaction generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ghL3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents IMAGIN-4D, a diffusion-based framework for image-guided 4D human-object interaction (HOI) generation. To improve fine-grained interaction control, the Spatially Factorized Image Encoder (SFIE) extracts supervised role-specific tokens for human pose, object pose, contact, and spatial relationships. A reference-frame localizer and window-gated conditioning provide temporal alignment, while the Frame-Aware Image Encoder (FAIE) retrieves frame-dependent visual cues. A role-aware conditioning mechanism integrates image, text, and waypoint information through separate conditioning pathways. Experiments on FBM and BEHAVE demonstrate improvements in image adherence, motion quality, and contact accuracy over image-conditioned baselines, although waypoint-following accuracy decreases.

Critical strengths

The paper addresses a meaningful limitation of HOI generation, as text and sparse trajectories alone cannot fully specify fine-grained interaction details. Using a reference image provides an intuitive way to control these details. The proposed supervised spatial factorization and frame-aware conditioning mechanisms are well motivated and complement each other. Experiments on FBM show substantial improvements in image adherence, motion quality, and contact accuracy, while the BEHAVE results provide additional evidence under real-image conditioning. Component ablations and image-flipping experiments further support the effectiveness of the proposed approach. The cross-domain and sketch-based evaluations also demonstrate the potential flexibility of the proposed image-conditioning framework.

Critical weaknesses

The evaluation does not fully support fine-grained interaction control, as the image-adherence metric omits reference-specific object orientation and hand-object contact details, which are not adequately captured by the separate contact metrics. IMAGIN-4D also shows higher waypoint errors, but the trade-off between image adherence and trajectory accuracy lacks systematic analysis. Although temporal comparisons and frame-offset experiments are provided, reference-frame localization accuracy remains unreported, and the contribution of frame-index conditioning in FAIE is not independently verified. Finally, while the component ablations use consistent settings, these differ from the final model, which additionally employs logit-normal diffusion-step sampling and hand-object coupling. Without isolating these training strategies, the contributions of the proposed architectural components to the final performance remain unclear.

Questions for authors

1.The image-adherence metric does not directly evaluate object orientation or hand-object contact details. Could the authors provide additional evaluations of these attributes and report image-adherence results for the image-free CHOIS baseline? 2.How accurately does the reference-frame localizer predict the target frame? Could the authors compare results using predicted and ground-truth reference frames and evaluate FAIE without frame-index conditioning to clarify its temporal contribution? 3.The ablation settings in Table 3 differ from those of the final model. Could the authors isolate the effects of the additional training strategies under consistent settings? A parameter-matched multi-token baseline without role-specific supervision would also help clarify the contribution of spatial factorization. 4.How does IMAGIN-4D compare with a simpler approach that predicts a 3D interaction state from the reference image and uses it for keyframe control? Could the authors also examine the trade-off between image adherence and waypoint accuracy under different guidance strengths?

Reviewer_yjjJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IMAGIN-4D for generating 4D HOI conditioned on text, object geometry, waypoints, and a reference image. The author propose several structural design to extract supervised tokens for pose, contact, and layout. The method achieve good results on FBM and BEHAVE comparing to the baseline.

Critical strengths

The spatio-temporal disentanglement against prior work is well motivated. The proposed image-adherence metrics effectively decouple spatial accuracy from temporal placement errors.

Critical weaknesses

The most critical weakness is that the setup of the task has strong bottleneck. It requires a known ground truth 3D object mesh, image conditioning, and waypoints, which is a huge pack for any applications. I feel like there should be more potential with the help of those powerful 2D generative models and we should make better use of them.

Questions for authors
  1. Given the strong visual priors we use, is it possible to relax any of the input constraints to generalize to more simple input pack setting?
  2. The generalization test (Figure 5) should be added to the supplementary video.
  3. Does training on synthetic SceneImg data suffice for general in-the-wild real image deployment, or is fine-tuning on real RGB data necessary to bridge the domain gap?
Reviewer_E9pKOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a pipeline for generating action sequences from input observations. It first uses SFIE to extract patch-level features and a Q-Former to capture information across four aspects. It then predicts correspondences with the reference and generates difference tokens, which are injected via AdaLN-Zero to guide action generation.

Critical strengths
  1. Interesting task formulation. The paper studies fine-grained control of human–object interaction using a single reference snapshot. This provides an intuitive way to specify pose, contact, and spatial relationships that are difficult to express through text and sparse waypoints alone.
  2. Well-motivated spatio-temporal conditioning. The combination of supervised spatial tokens and frame-aware visual tokens captures distinct interaction cues while adapting their influence across the generated sequence.
  3. Role-aware conditioning design. Separate AdaLN streams for spatial tokens, waypoints, and text, together with cross-attention for frame-aware tokens, provide a structured mechanism for integrating heterogeneous conditions. The reported results demonstrate improved reference adherence and motion quality.
  4. Useful data pipeline which can be useful for following works.
Critical weaknesses
  1. Limited validation of FAIE’s temporal specialization. FAIE uses frame- and text-conditioned queries to extract tokens from a single reference image, with supervision provided indirectly through motion denoising. However, it is unclear whether FAIE extracts useful frame-specific information. Attention visualizations and quantitative ablations without frame-index conditioning would help verify this.

  2. It would be helpful to compare other image encoders to see whether they improve the results. The supp only compares the similarity.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.