acceptodds
Withdrawn from ICLR 2027

PACHO: Parallel Affordance and Contact Diffusion for Human-Object Interaction Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Text-driven 3D human-object interaction (HOI) synthesis requires precise spatiotemporal synchronization and stable physical contact between the human and object. Existing approaches struggle to capture the spatiotemporal co-evolution of localized human-contact and object-affordance regions, leading to loose physical coupling. To provide fine-grained spatial guidance, our key insight is to explicitly model the dynamic “handshake” between entities: the tightly coupled, time-varying human contact regions and functional object affordances. We propose , a novel diffusion framework that predicts object affordance and human contact maps in parallel to guide human–object motion denoising. Specifically, PACHO utilizes Dual Motion-Prior Learners to extract decoupled kinematic representations for both the human and the object. To bridge these independent streams, we introduce a Cross-Affordance Interaction module that performs adaptive, part-aware point-level contact-affordance reasoning to establish a contact-aware bidirectional physical coupling. During denoising, a joint motion-salience objective supervises the motion alongside the affordance and contact, encouraging local geometric consistency. Extensive experiments demonstrate that PACHO achieves state-of-the-art performance, with clear improvements in physical realism, interpenetration reduction and contact stability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Human-Object Interaction, Human Motion Generation, Diffusion Models, Affordance Learning, Contact Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t9n4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PACHO, a framework for text-driven 3D human-object interaction synthesis, which introduces a cross-affordance interaction module that jointly models human contact and object affordance throughout the motion generation process. Experiments on the InterAct and OMOMO benchmarks demonstrate improvements in motion quality and contact-related metrics.

Critical strengths
  • The joint object-affordance and human-contact prediction module is well motivated and carefully designed, providing a more comprehensive representation than previous approaches that predict only a single affordance map on whole-body HOI.

  • The proposed method achieves state-of-the-art performance on the evaluated benchmarks.

Critical weaknesses
  • The novelty of the proposed method is somewhat limited, as prior works [1, 2] have explored similar ideas involving temporal affordance modeling. Additionally, it would be valuable to adapt the method in [1] to the whole-body HOI generation setting and include it as a baseline for comparison.
  • How does the proposed method perform on the GRAB dataset, where the affordance regions typically remain relatively static throughout an interaction sequence?

[1]. Song, Mingyeong, et al. "JointHOI: Jointly Generating Contact Maps Enhances Hand–Object Interaction Generation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026. [2] Zhang, Wanyue, et al. "Bimart: A unified approach for the synthesis of 3d bimanual interaction with articulated objects." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

Listed in the weaknesses

Reviewer_aDNvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PACHO tackles the problem of text-conditioned human-object interaction (HOI) synthesis. The proposed method first predicts human (hh) and object (oo) motion features separately via two decoupled modules, then each stream predicts a part-aware per-point salience on the corresponding surface (human or object mesh). The key idea of the proposed method is to couple human and object streams via a cross-attention module, where human motion features attend to object affordance map (o→ho \rightarrow h) and vice versa object motion features attend to human contact map ( h→oh\rightarrow o ), afterwards separate decoders predict the motion. PACHO is evaluated on OMOMO and InterAct benchmarks outperforming baselines on most metrics with the largest gaps in contact distance and penetration.

Critical strengths
  • The method is clearly motivated. The dense bi-directional contacts are a clear improvement over traditional human->object contacts.
  • Thorough ablation study. The experiments extensively cover the method components and design choices.
Critical weaknesses

The quality of the evaluation is the most critical weakness of the work

  1. Consistency of train-test split. The split is claimed to be borrowed from LIGHT, which according to its GitHub repo is different from the original OMOMO split and uses sequence-level split instead of holdout subjects. This raises multiple concerns:
    • 1.1. The baselines in Table 1 (OMOMO) use different splits. The values for OCD and FSR of MDM, InterDiff and ChainHOI exactly match (including +- values) the ones in the ChainHOI paper, which uses a third type of split (holdout objects), while FID, R-prec and Pene numbers do not match either ChainHOI or LIGHT papers.
    • 1.2. Claim of "unseen wooden chair" in Figure 8 is not supported by the split: if the LIGHT's split was used then this object was seen during training.
  2. Consistency of learned evaluator (for FID and R-prec). LIGHT's row in Table 1 (OMOMO) is copied from the original paper (Table D), yet the number for R-prec of PACHO exceeds the corresponding GT value in the original table and the "Ground Truth" row is not present in PACHO's table. So either the R-prec number is incorrect or a different evaluator was used, making numbers incompatible. At the same time, InterDiff results do not match the Table D of LIGHT with significant gap (FID 0.234 vs 0.163, R-prec 0.129 vs 0.312).
  3. Baseline numbers appear to be copied, not re-evaluated as stated. Appendix D claims the fair evaluation of the baselines, yet most of the numbers (e.g. Table 1 InterAct part is a copy of Table 1 from LIGHT) appear to be copied from the corresponding works (stds match), which raises several concerns:
    • 3.1. Whether the same learned evaluator is used for all the methods. This is a required condition for FID and R-prec numbers to be compatible.
    • 3.2. Which of the numbers were copied, and which of the methods were actually re-evaluated.
  4. The details of averaging across multiple runs are not provided (what is the +- values, how many runs are averaged).

Other weaknesses:

  1. The key design choice is not properly ablated. PACHO uses only salience maps to exchange information between human and object streams, instead of letting human and object motion features cross-attend to each other. However, this choice is motivated only by the assertion of "premature entanglement" (Lines 119-120). The only ablation, which appears to go in this direction, is "Direct Motion Interaction" in Table 10, but no description of this method variant is provided.
  2. Lack of formal definition of some method components. For example, ZthZ^h_t and ZtoZ^o_t are described as "noisy human and object motion tokens in latent space" (Line 215), however the latent encoder and / or format is not formally defined. Similarly, Lines 246-248 introduce "interaction tokens" that come from "lightweight point encoders", which are also not formally defined (e.g. what is the input to these encoders, are the tokens per-point or per-frame, etc.). Finally, Figure 2 has a single Motion Generation block, whereas the text says that human and object motion are decoded separately.
  3. Limited video results. The video presents results for only 2 sequences, all the other results are shown as sparse frame renderings.
  4. The supplementary video is not updated from the NeurIPS submission (Submission ID and NeurIPS logo on all slides).
Questions for authors

Could the authors please clarify the gaps in the evaluation mentioned above, namely:

  1. Which OMOMO split is used and is it the same for every row of Table 1?
  2. Which learned evaluator is used for FID and R-prec (LIGHT's checkpoint or newly trained) and is the evaluator same for all the evaluated methods?
  3. Which of the baselines were re-trained and / or re-evaluated?
  4. How are the numbers across multiple runs averaged in the tables and how many runs are performed?
  5. What is the definition of "Direct Motion Interaction" method variant?
Reviewer_Wc8EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes doing parallel diffusion for human motion and object motion, predicts contact maps (affordance for objects) and then applies bidirectional cross-attention between them for denoising a joint interaction motion.

Critical strengths
  1. Importance of modeling dynamic human-object contact: The motivation for generating time-varying contact-relevant regions on both the human and object surfaces is sound and the method is designed with this objective in mind, making it a sensible approach to HOI synthesis.

  2. Extensive ablation studies supporting most design decisions: The paper provides a comprehensive set of ablations covering the dual motion-prior architecture, bidirectional attention, contact-affordance co-prediction, representation granularity, and geometric alignment supervision. These experiments provide evidence of the effectiveness of the individual components and their combination.

Critical weaknesses
  1. The authors evaluate the effect of contact map predictions, but not their correctness: There is a visualization of the temporal evolution of contact maps in Figure 4, and it is demonstrated that removing these saliency predictions degrades generation quality. However, there is no direct quantitative evaluation of whether the model predicts accurate contact or affordance maps on held-out samples.

  2. An explicit point-to-point contact baseline would strengthen 'handshake' the argument: The formulation of this method uses the cross-attention between saliency maps and motion features to do the semantic 'handshake' and while this implicitly learns contact patterns between the objects and the human, this design choice should be validated against a baseline that explicitly represents point-to-point contact correspondences.

  3. Underspecification of details and confusing writing: Several important implementation details remain unclear. Some examples are - 1. The construction of interaction tokens in CrossAfford (Eq. 4) is not specified, including their spatial and temporal resolution and whether point coordinates are encoded. 2. The object part embedding mechanism (Eq. 2) is also unclear, particularly how arbitrary part identifiers from P3-SAM are represented consistently across objects. 3. How does F_c condition the transformer - Is it adaLN, an extra token, or addition? 4. What is global salience and how does it differ from part-aware and point-level saliences? 5. The adaptations to baselines like hands-only Text2HOI and CHOIS are also not clearly stated. 6. Further, why does the baselines set change based on the dataset? If App. D says baselines were retrained or evaluated “on the same benchmark splits”, why do the tables simply inherit baselines and numbers from previous papers instead of having the same baselines for both methods?

The term 'saliency' is used everywhere making the reading experience confusing. It essentially refers to a distance-based contact map that is normalized [0,1]. I also don't understand why the authors keep referring to the the human and object contact maps with separate terms especially when they are both derived from the same distance metric. 'Affordance' maps for objects typically refer to action possibilities and are dependent on function. The word 'adaptive' is also mentioned a couple times but I never understood what part of the method it refers to.

Questions for authors
  1. Can you verify the accuracy of predicted contact maps?

  2. Can you compare against a baseline where the motion prior remains the same but the method predicts correspondences between human and object surface points and uses these correspondences to guide cross-attention?

  3. Please specify the missing implementation and baseline adaptation details

  4. I would also like to know which numbers came from models that were actually trained from scratch and which ones were inherited from tables in prior work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.