acceptodds
Under review as a conference paper at ICLR 2027

Surface-Sync Human–Object Interaction Motion Generation via Contact Modeling and Joint Hand–Body Optimization

Abstract

Human–object interaction (HOI) motion generation remains challenging because realistic interactions require modeling not only human and object motions, but also their evolving physical relationship at the interaction interface. Existing approaches mainly represent interactions through joints, trajectories, or object poses, leaving surface correspondence, contact quality, and mechanical stability implicit. In this paper, we propose Surface-Sync, a unified framework for HOI motion generation that explicitly models interaction from representation and physical reasoning perspectives. We introduce a surface-centric interaction representation based on sparse surface markers, which captures dynamic human–object surface correspondences through contact activation and canonical object-surface anchors. This representation provides a unified basis for incorporating geometric compatibility and wrench-space physical priors, enabling the generation of physically stable and coherent interactions. Furthermore, we propose an interaction-driven whole-body coordination mechanism that propagates local interaction constraints from end-effectors to the surrounding body structure, improving the consistency between manipulation and global motion. Extensive experiments demonstrate that our approach generates more realistic, physically plausible, and coordinated human–object interaction motions compared with existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D motion generation, human motion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VrdLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents surface-sync, a method for generating human–object interaction motions from text and object geometry. The main idea is to explicitly model where and how the human and object surfaces make contact. Built on a dual-branch diffusion model, it jointly predicts surface-marker motions, contact activation, and contact locations on the object. It adds objectives for surface alignment, contact coherence, and force-closure-inspired stability, and other constraints. The authors evaluate the method on InterAct and test cross-dataset transfer. The reported results show improvements over the compared baselines in motion quality, text alignment, and several contact-related metrics. Ablation studies and qualitative examples further support the contribution of the main components.

Critical strengths
  1. The model predicts both whether contact occurs and where it occurs on the object surface, together with human and object motion. This gives the geometric and stability objectives a shared representation to work with. I see this as the paper’s main strength: it directly addresses interaction quality, which plausible-looking motion trajectories alone do not ensure.

  2. On InterAct, the method improves FID, text alignment, contact distance, and realism over the compared baselines. Cross-dataset results also show consistent gains over HOI-Diff and InterDiff, while the ablations support the usefulness of the representation, physical objectives, and body coordination. Together, these results make the proposed design more convincing than qualitative examples alone would.

Critical weaknesses
  1. The evidence is limited to relatively simple interaction settings. The formulation focuses on one person interacting with one object. It is unclear how it would handle multiple objects or people, where several human–object and object–object contacts must be coordinated. The video examples also appear to use short, single-action instructions, leaving performance on longer, multi-step instructions untested. These are common scenarios, and the paper does not explain or demonstrate how its contact representation and coordination objectives would extend to them. This limits how broadly I can interpret the claimed generality of the framework.

  2. The visual results still show noticeable contact errors and low-quality motion. In the supplementary videos, I observed hand–object misalignment, sliding, and contact at incorrect locations. These issues directly concern the paper’s central claim of improving surface contact and interaction stability. Although the quantitative results are encouraging, the remaining artifacts make it unclear how reliably the proposed methods can be used in real applications.

Questions for authors
  1. Which parts of the current formulation would carry over to scenes with multiple people and objects, and which would require substantial changes? In particular, how would it represent and coordinate simultaneous human–object and object–object contacts? This matters to my assessment of its practical applicability beyond the single-person, single-object setting.
  2. In my own work, I have found that a VLM-guided refinement loop. Rendering a 3D scene in Blender, asking a model such as Qwen (or even Astra) to identify semantic and physical inconsistencies, and iteratively adjusting the motion can substantially improve physical plausibility. The results are even better than yours shown in the supplementary video. What advantages does your specialized formulation offer over such an approach, for example in contact accuracy, temporal consistency, computational cost, or reliability? A controlled comparison or concrete evidence would help justify the added complexity of the method.
  3. Can the current model follow a multi-step instruction, such as “pick up the object, turn around while holding it, and place it on a nearby surface,” starting from a specified initial human–object configuration? Please clarify whether initial-state conditioning is supported and, if so, provide a representative result showing contact transitions and continuity between actions. This would help establish whether the method extends beyond the short, single-action examples shown.
Reviewer_FmzWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the generation of 3D human-object interaction based on text and object geometry. The method uses a two-branch diffusion framework to generate human and object motions represented by surface marker points, and predicts the contact activation probability of hand marker points and their corresponding positions on the surface of the object. The author interacts with loss constraints inspired by surface normal, curvature, contact area consistency and force screw space, and uses the loss of structural distance between human marker points to promote the coordination of hand and body movements. In InterAct and cross-dataset settings, the paper reports action quality, contact and penetration indicators, ablation experiments, and user studies.

Critical strengths

(1) Research questions are valuable. Human-object interaction requires not only reasonable human and object trajectories, but also coordination between the two during contact. The contact gap, penetration and object movement are not synchronized with the hand movement, which will directly affect the credibility of the generated results. In this paper, the surface contact relationship changing with time is explicitly modeled, and the research goal is clear and meaningful.

(2) The proposed method shows effectiveness. In the InterAct experiment, this paper achieved the best FID and R @ 3, and the contact distance and contact ratio were also better than the listed generation baselines. Component ablation and user research further support the role of surface contact representation, physical heuristic constraints, and body coordination constraints.

Critical weaknesses

(1) There is a lack of explanation for fair comparison of different baselines. Table 1 evaluates CHOIS, InterDiff, Text2HOI and other methods with this article, but these methods originally have different generation tasks or input conditions. For example, CHOIS uses the initial state and sparse object path conditions, InterDiff faces future interaction prediction, and Text2HOI mainly generates hand-object interaction. The paper has not yet explained how each method adapts to the geometric conditions of the text and objects in this paper, whether it is retrained, and whether the data division and index calculation are consistent.

(2) The direct evidence for contact quality and physical rationality is still insufficient. The cross-qualitative comparison in Figure 3 and the supplementary video only compares the HOI-Diff, and there is no comparison of other similar methods. In addition, in some of the results of this paper in the supplementary video, more serious wear or insufficient contact can still be observed. These phenomena are precisely the core issues that the paper attempts to solve.

Questions for authors

(1) What is the added value of this paper compared to CG-HOI ? CG-HOI also jointly generates human body, object and contact information under text and object geometry conditions, and uses contact guidance to generate. Does the new method proposed in this paper bring the ability or effect different from CG-HOI ?

(2) How does the generated object remain rigid ? In this paper, the object motion is expressed as the position and velocity of multiple surface marker points. In the process of generating or after generating, is there any method to ensure that the relative position of each marker point on the same object remains unchanged ?

(3) What does the original motion in Equation ( 6 ) come from in the reasoning stage ?

Reviewer_W5PoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Surface-Sync, a framework for modeling human-object interactions by jointly generating human and object motions, contact states, and corresponding locations on object surfaces. Building on the HOI-Diff backbone, the method introduces a surface marker representation and predicts contact states and corresponding object-surface anchors for a subset of human markers located on the hands. The authors further introduce losses for surface alignment, contact consistency, force-closure-inspired stability, and structural preservation of distances between body markers. The authors claim that these designs promote continuous surface contact between humans and objects, physical stability, and interaction-driven whole-body coordination.

Critical strengths

Jointly generating contact states and corresponding locations on object surfaces alongside motion is a meaningful direction, as it explicitly models the relationship between humans and objects. However, the conceptual merit of this direction should be distinguished from whether the proposed method successfully realizes it in practice.

Critical weaknesses
  1. Significant gaps between the stated objectives and the proposed methodology

1-1. A substantial gap between the goal of modeling fine-grained surface interactions and the proposed sparse representation-based approach.

The authors claim to model surface-level interactions through contact correspondences and local geometric constraints defined on sparse surface markers. However, it remains unclear how these constraints ensure proper contact between the hand and object surfaces in regions without marker-correspondence pairs. In particular, even if the surface normals and curvatures of the hand and object are properly aligned at some correspondence pairs, these constraints only enforce local geometric relationships around those pairs, without directly constraining the contact relationships between the hand and object surfaces in other regions. Therefore, such constraints alone are insufficient to establish that the hand and object surfaces form appropriate contact or satisfy non-penetration conditions in those regions. This raises a fundamental question about the extent to which local contact constraints defined on sparse markers can actually ensure the continuous and consistent surface-level contact emphasized throughout the paper.

1-2. A substantial gap between the goal of interaction-driven whole-body coordination and the proposed structural coordination objective.

The structural coordination objective introduced in Eq. (6) penalizes deviations from reference distances between structurally connected body markers. While this term may help prevent structural distortions and introduce coupling between connected markers, it does not necessarily imply that the movements of the arms, shoulders, and torso are appropriately coordinated in response to hand-contact requirements.

In Figure 1(c), the authors contrast an unnatural posture involving excessive arm extension with a more natural posture involving torso inclination. However, it is unclear how Eq. (6) distinguishes between these two postures or encourages a whole-body configuration that better accommodates the contact requirements. The authors do not sufficiently explain or independently validate the important distinction between preserving inter-marker distances and achieving interaction-driven whole-body coordination.

  1. The qualitative results do not sufficiently demonstrate whether the paper's stated objectives have been achieved.

Aside from the badminton racket grasping and swinging motion shown in the supplementary material, the qualitative examples do not convincingly demonstrate accurate and sustained surface contact between the hands and objects, physically plausible motion resulting from force-closure-inspired stability, or effective whole-body coordination. Consequently, the presented qualitative evidence is insufficient to establish that the proposed method achieves its central objectives.

Questions for authors

Please refer to the Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.