acceptodds
Withdrawn from ICLR 2027

PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-conditioned full-body human–object interaction (HOI) generation requires synthesizing human motion and object trajectories that match the input text while remaining precisely coordinated over time. Most methods represent the human and object as separate trajectories and predict the global human-object couplings. Learning this complex, dynamically changing relationship implicitly, however, often yields object drift, missed contact, and penetration. We introduce PAMI, a Part-Anchored Motion framework for Interaction generation. Inspired by the classic Hough Transform, our key idea is to localize object motion by letting body-part anchors vote for it: we express object motion relative to multiple body-part anchors and use PamiVAE to learn an interaction latent space, decoding frame-wise weights that aggregate these part-specific votes. Building on this representation, PAMI generates interactions in a coarse-to-fine hierarchy. PamiGen first generates a coarse human–object interaction from text in this structured latent space, and PamiRefiner then recursively resolves fine-grained contact geometry using a hybrid surface-sensing representation, combining long-range probes that capture overall body-part influence with short-range sensors that resolve detailed contacts near the object surface. Experiments on InterAct show that PAMI generates more faithful interactions and more accurate human-relative object motion than previous methods, achieving 14.5% higher contact recall than the previous state of the art. Extensive ablations validate the contributions of both the part-anchored voting representation and hybrid surface-sensing refinement.

Primary area:
generative models
Keywords:
human-object interaction generation, text-to-motion generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JALcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new method named PAMI for text-conditioned full-body human–object interaction generation. The core contributions of this paper are the proposed part-based latent generative framework and the body-part-based object voting representation. The proposed representation anchors object motion to body parts and lets the anchors vote for object movement via learned weights, achieved by PamiVAE. Then, the authors propose a refinement network that corrects the contact details using surface probes around the body and fingers. The authors conduct experiments on the InterAct dataset to validate the effectiveness of the proposed method.

Critical strengths
  1. The anchor-relative object representation seems effective. Experimental results show that removing the anchors clearly degrades contact quality even before refinement.
  2. The proposed refiner can be easily applied to other generators without retraining and improves their interaction quality.
Critical weaknesses
  1. The technical contribution is limited. As introduced in the related work section (lines 141–143), learned weights over wrist-relative and free object trajectories were already proposed in Dex2HOI. The body-part partition follows FrankenMotion, and the surface probes follow GRAFT and GEARS.
  • The main extension is adding root and ankle anchors, but this brings almost no gain before refinement, as shown in Table 7 (w/o leg and root anchors).
  1. Most of the improvement in interaction quality comes from the refiner, while the baselines in Table 1 have no such post-processing.
  • Before refinement, PAMI is close to LIGHT on hand contact recall (0.758 vs. 0.730). The claimed 14.5% gain only appears after refinement.
  • When the same refiner is applied to LIGHT, the gap becomes much smaller, as shown in Table 9.
  • The authors should discuss why PAMI has the worst foot skating ratio in Table 1.
  1. The evaluation protocol is unclear.
  • The same baselines have different results in Table 1 and Table 9. For example, the FSR of LIGHT is 0.050 in Table 1 but 0.105 in Table 9.
  • The paper claims to follow LIGHT, but uses a different R-Precision batch size and obtains different ground-truth values.
Questions for authors

See Weaknesses.

3 comments

1. Technical contribution

Technical contribution is limited. Our contributions are two fold: 1). a part-anchored generative framework that uses anchor-relative object representation to learn a compact latent space for interaction and 2). A hierarchical generation framework where PamiGen first generate coarse motion in latent space and then PamiRefiner further enhances geometric correctness in explicit space. Our method is inspired by Dex2HOI, FrankenMotion, GRAFT/GEARS but differs significantly from them.

1). Distinction from Dex2HOI and FrankenMotion: Our part-relative representation is inspired by Dex2HOI, but extends wrist-based anchoring to full-body anchors. We adopt the body-part partition from FrankenMotion and design a part-anchored latent space for coarse human-object interaction generation. In contrast, both Dex2HOI and FrankenMotion generate motion in explicit geometric space. Table 7 shows that our part factorization improves R@1 and FID before refinement. Specifically, our part-factored VAE reduces reconstruction MPJPE from 38.8 to 15.2 mm (Table 6). In the full pipeline, part factorization improves FID from 0.171 to 0.092 and full-body contact F1 from 0.327 to 0.457 (Table 2). Our PamiRefiner is another novel component that differs from existing methods and further improves interaction quality, as discussed below.

2). Distinction to GEARS/GRAFT: Our PamiRefiner draws inspiration from the geometric sensing representation from GRAFT and GEARS yet we need to address the challenges introduced in text to HOI generation. GRAFT is designed for single-frame human-scene refinement where the scene is static while PAMI needs to deal with temporally dynamic human-object interaction. GEARS assumes high quality hand and object trajectory from mocap or multi-view captures and generates the finger poses while our PamiRefiner focuses on improving finger contacts from imperfect hand-object trajectories. Naively combining GRAFT and GEARS leads to large motion jitters and finger penetrations as GRAFT is not temporal-aware and GEARS is not robust to noisy wrist trajectories.

We train GRAFT and GEARS on the same InterAct training set following the same network and training recipe. To compare with PamiRefiner, we first run GRAFT on the output of PamiGen to refine body and wrists and then run GEARS to replace fingers. The comparison is reported in table below. We can see that simple combination of GRAFT and GEARS leads to worse FID and FSR due to motion jitter, worse penetration and hand contact due to inaccurate finger generation. Our PamiRefiner harmonizes the hybrid geometry sensing probes in a new spatiotemporal transformer network and is trained with our proposed two-stream data together with additional geometric loss designs, leading to better performance across all metrics. We will add this ablation to table 3.

Refinement methodR@1 ↑FID ↓FSR ↓Pene ↓Contact →Body F1 ↑Hand F1 ↑
GRAFT + GEARS adaptation0.8400.1170.0770.1030.1760.4290.808
PamiRefiner0.8400.0920.0370.0620.1930.4570.821

Root and ankle anchors brings no gain before refinement. This is due to the heavy imbalance in test set: only 105 of 907 test sequences contain non-hand contact in at least 10% of ground-truth frames, where the body anchor actually plays a role. Hence average over full test set measures mainly the effectiveness of hand anchors. We hence evaluate the contact scores of the body parts corresponding to our additional anchors on these 105 sequences, see table below. Our F1 improves from 0.267 to 0.302, clearly demonstrating the effectiveness of our method. We would like to emphasize that our contribution is not simply “extending the anchors”. Our method also contributes the part latent generation and PamiRefiner as discussed above.

Variant (before refinement)Body P ↑Body R ↑Body F1 ↑
No anchors0.2760.2720.239
Wrist only0.2970.3000.267
Full anchors0.3260.3490.302

2. Contribution of the refiner and fairness of baseline comparisons

Interaction improvement comes from the refiner while baselines don’t have that. We want to note that all baselines have their own geometry-improvement mechanisms: HOI-Diff (affordance-guided correction), CHOIS (geometric guidance), InterDiff (kinematics-informed iterative correction), and LIGHT (contact-aware guidance). It is hence unfair to compare part of our method (table 7) with the full method of baselines (table 1). Without PamiRefiner, our recall improvement over LIGHT (0.730->0.758) is still significantly better than LIGHT vs. the previous best InterDiff (0.718->0.730). This clearly shows the contribution of our part-anchored latent space formulation.

LIGHT + the same refiner has smaller gap to PAMI. Applying the frozen PamiRefiner to LIGHT leads to improvement of hand-contact F1 from 0.704 to 0.755 and full-body F1 from 0.328 to 0.408 (Table 9). Yet our complete PAMI remains significantly better than LIGHT + PamiRefiner on both measures (0.821 for hands and 0.457 for full body). LIGHT benefits from our PamiRefiner demonstrates the generalization capability of our method yet our integral design of hierarchical generation still has superior performance gain.

Foot skating is worse. We thank the reviewer for pointing this problem and revisit our evaluation code. We find there is one inconsistency in the definition of ground plane which affects FSR significantly. LIGHT defines ground plane as the lowest point of human and object meshes while our previous evaluation used the lowest point of body joints. After aligning the protocol with LIGHT, our FSR improved from 0.078 to 0.037, which is better than LIGHT (0.050) and close to the best Text2HOI (0.035). The evaluation of baselines and other metrics are unaffected. We will update this number to table 1.

3. Evaluation protocol and metric discrepancies

Why Table 1 and Table 9 differ. LIGHT outputs 3D body joint positions with separately modeled hand rotations; InterAct uses body-surface-marker trajectories. Neither directly provides the SMPL pose parameters θ needed by PamiRefiner, which predicts pose parameter updates Δθ on body and object poses. We use inverse kinematics (IK) to convert outputs of LIGHT and InterAct before applying the frozen refiner, as stated in the original submission (Lines 470–473). Table 1 evaluates original baseline outputs while Table 9 reports the baseline results after IK. IK can change measured metrics, including LIGHT FSR (0.050 in Table 1 vs. 0.105 in Table 9). Note that the FSR evaluation problem above does not affect this as the evaluation uses LIGHT’s own code. Each within-Table-9 comparison uses the same conversion, ensuring a consistent before/after comparison. We will explain this further in table 9.

LIGHT evaluation protocol. The publicly released evaluation code of LIGHT contains one problem where all contact metrics consider only the final batch instead of the full test set. We contacted the authors and they rerun the evaluation with their saved baseline results again but using a different batch size of 64. This is different from their original paper where 256 was used. We simply copied the new table they sent us and evaluate our method following their new protocol. Since batch size affects R-precision, FID, and contact metrics, penetration and FSR are recalculated using corrected code, hence the GT values we reported differ from the public version. The authors promised that they will update the new table to arXiv and we will also clarify this better in Sec. 4.

Dear Reviewer JALc,

Thank you again for your detailed review. As the discussion period is coming to a close, we would be grateful to know whether our responses address your concerns, or if there is anything else we can clarify.

Best regards, The Authors

Reviewer_2vjCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latent representation for modeling fine-grained human-object interactions, incorporating object coordinates and weights relative to individual human body parts. Conditioned on text and object geometry encoded using BPS, the method generates latent representation via rectified flow and subsequently refines the generated results using PamiRefiner. The proposed method outperforms the compared baselines in both semantic metrics, such as FID, and physical interaction metrics, such as contact and penetration.

Critical strengths

The proposed method achieves better quantitative results than the compared baselines. In particular, its strong performance in FID and R-precision suggests that the method generates motions that are not only physically plausible but also semantically coherent and well aligned with the given text and object conditions.

The proposed method appears well motivated and technically sound. The anchor-based representation seems suitable for capturing the approximate spatial relationships between humans and objects. Furthermore, since object motion in HOI sequences is closely coupled with the movements of individual body parts, the proposed representation appears well suited to modeling such dependencies.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification of the proposed extensions.

My primary concern is the limited novelty of the proposed methodology. First, the 'body-part based object voting representation', presented as a key idea of the paper, has already been explored in prior works such as CG-HOI and Dex2HOI, as acknowledged by the authors. Extending this mechanism to full-body part anchors alone does not appear to constitute a substantial methodological contribution. Moreover, the paper lacks sufficient qualitative explanations and supporting evidence regarding when and why the full-body anchor formulation leads to better performance. From a quantitative perspective, Table 7 in the appendix also indicates that the full-anchor formulation does not consistently improve performance. Second, the refinement designs of PamiRefiner are closely related to those of GRAFT and GEARS. It remains unclear what new technical challenges arise when extending these designs to full-body HOI refinement, how the proposed method addresses these challenges, and which additional design choices are essential to doing so. Furthermore, the integration of PamiRefiner with the proposed representation does not appear to introduce a clearly identifiable methodological contribution beyond combining existing techniques.

  1. Insufficient evidence that the authors' methodological contributions address the claimed limitations of prior methods.

Without PamiRefiner, the anchor-based representation contributes to improved contact modeling but does not consistently outperform existing methods in terms of contact ratio and penetration. In particular, the substantial improvement in penetration appears primarily after applying PamiRefiner. Given the limited methodological novelty of PamiRefiner discussed above, the performance of the full Pami pipeline alone does not provide sufficient evidence that the proposed representation effectively addresses the limitations of prior methods identified by the authors, particularly missed contacts and penetration.

Questions for authors

Please refer to the Critical Weaknesses.

2 comments

1. Methodological novelty and justification

Methodological contribution is limited. Our part relative representation is similar to Dex2HOI but our contribution is not limited to this representation. We also introduce a part-factored generative latent space for semantically aligned interaction generation and PamiRefiner to enhance geometric correctness. In contrast, both Dex2HOI and CG-HOI perform generation directly in explicitly geometry space. The contribution of our part latent space is well evaluated in our ablations: Our part-factored VAE reduces reconstruction MPJPE from 38.8 to 15.2 mm (Table 6). In the full pipeline, part factorization improves FID from 0.171 to 0.092 and full-body contact F1 from 0.327 to 0.457 (Table 2). Our PamiRefiner also further improves the fine-grained interaction quality as ablated in table 3. We will make this clearer in introduction L76-86.

Improvement introduced by full-body anchors is relatively small.

This is due to the heavy imbalance in test set: only 105 of 907 test sequences contain non-hand contact in at least 10% of ground-truth frames, where the body anchor actually plays a role. Hence average over full test set measures mainly the effectiveness of hand anchors. We hence evaluate the contact scores of the body parts corresponding to our additional anchors on these 105 sequences, see table below. Our F1 improves from 0.267 to 0.302, clearly demonstrating the effectiveness of our method.

Variant (before refinement)Body P ↑Body R ↑Body F1 ↑
No anchors0.2760.2720.239
Wrist only0.2970.3000.267
Full anchors0.3260.3490.302

PamiRefiner vs. GRAFT / GEARS. We draw inspiration from their geometric sensing representation but we have several important design changes that distinguish us from theirs. GRAFT is designed for single-frame human-scene refinement where the scene is static while PAMI needs to deal with temporally dynamic human-object interaction. GEARS assumes high quality hand and object trajectory from mocap or multi-view captures and generates the finger poses while our PamiRefiner focuses on improving finger contacts from imperfect hand-object trajectories. Naively combining GRAFT and GEARS leads to large motion jitters and finger penetrations as GRAFT is not temporal-aware and GEARS is not robust to noisy wrist trajectories.

We train GRAFT and GEARS on the same InterAct training set following the same network and training recipe. To compare with PamiRefiner, we first run GRAFT on the output of PamiGen to refine body and wrists and then run GEARS to replace fingers. The comparison is reported in table below. We can see that simple combination of GRAFT and GEARS leads to worse FID and FSR due to motion jitter, worse penetration and hand contact due to inaccurate finger generation. Our PamiRefiner harmonizes the hybrid geometry sensing probes in a new spatiotemporal transformer network and is trained with our proposed two-stream data together with additional geometric loss designs, leading to better performance across all metrics. We will add this ablation to table 3.

Refinement methodR@1 ↑FID ↓FSR ↓Pene ↓Contact →Body F1 ↑Hand F1 ↑
GRAFT + GEARS adaptation0.8400.1170.0770.1030.1760.4290.808
PamiRefiner0.8400.0920.0370.0620.1930.4570.821

2. Evidence that the contributions address prior limitations

Contributions do not address the previous limitations. We would like to emphasize that our contribution is not limited to anchor-based representation, as discussed in our reply above. Our contributions include part-factored latent generation PamiGen and interaction refinement module PamiRefiner. PamiGen without PamiRefiner is not claimed to eliminate fine-grained contact or penetration errors (Lines 84–94, 183–187). Its role is to generate semantically coherent global motion and coarse human–object coordination in a latent space. Before refinement, PamiGen significantly improves FID (0.103 vs. 0.153) and Top-1 R-precision (0.839 vs. 0.719) relative to LIGHT, due to our structured latent space.

Comparing this result (Table 7), the output of our partial method, directly with the full method of baselines (Table 1) is however unfair as we also have PamiRefiner, another contribution that is specifically designed to improve fine-grained interaction geometry through iterative surface-aware correction (Lines 286–290). Both stages are explicit contributions (Lines 101–111) and are integral parts of our full method. Combining both contributions, our full PAMI method addresses the limitations of previous method as shown in table 1.

Dear Reviewer 2vjC,

Thank you again for your detailed review. As the discussion period is coming to a close, we would be grateful to know whether our responses address your concerns, or if there is anything else we can clarify.

Best regards, The Authors

Reviewer_67yYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PAMI, a framework for text-conditioned full-body human-object interaction (HOI) generation. The framework represents object motion through a set of body-part-relative translations, which are aggregated using learned per-frame voting weights. This representation is embedded into a part-factored interaction latent space using PamiVAE. A text-conditioned rectified-flow model, PamiGen, generates coarse human-object interactions in this space, followed by PamiRefiner, which employs hybrid long- and short-range surface probes to improve contact geometry through iterative refinement.

Critical strengths
  • Well-motivated representation for human-object coupling. The proposed part-anchored representation provides an intuitive geometric inductive bias for modeling interactions.
  • Effective geometric refinement and convincing ablations. The coarse-to-fine framework is well motivated, and the hybrid surface-sensing mechanism provides complementary geometric information at different spatial scales.
  • State-of-the-art results are reported
Critical weaknesses
  • The body part anchoring is somewhat similar to the explicit joint and chain level graph interation. Could you please compare with this graph interaction modeling with your part-achor voting? I would appreciate if any quantitative comparison could be provided.
  • Could authors please provide some quantitative evaluation and ablations on the GRAB subset. Would the body-part anchor module be effective under this less dynamic, but more precise whole-body dexterous manipulation setting? Also it would be good to compare with some SOTA methods on dexterous manipulation [1].

[1] Pi, Huaijin, et al. "Coda: Coordinated diffusion noise optimization for whole-body manipulation of articulated objects." Advances in Neural Information Processing Systems 38 (2026): 31758-31780.

Questions for authors

Listed in the weaknesses

2 comments

Q1. Joint-/chain-level graph interaction

Thank you for the suggestion. Could the reviewer confirm whether the intended reference is ChainHOI (CVPR 2025) or another graph-interaction method? ChainHOI models joint- and chain-level graph interactions, whereas PAMI represents object translation in body-part local frames and composes those anchor predictions using learned per-frame weights within a part-factored generative latent space. To our knowledge, ChainHOI has not released code or pretrained models, and its reported evaluation protocols also differ from InterAct and HOI-Diff. Hence, a quantitative comparison would require significant efforts for reproducing the whole method.

Q2. GRAB subset evaluation and CoDA comparison

GRAB testset ablations. We evaluate whether part-anchored generation remains effective for precise whole-body dexterous manipulation. Full PAMI with anchors achieves hand contact F1 of 0.905 versus 0.882 without leg/root anchors and 0.714 without any anchors. Removing part factorization worsens FID from 0.547 to 0.820. These results show that our full-body anchors and part-factored modeling are also effective for whole-body dexterous manipulation.

Table A. PAMI ablations on GRAB testset

VariantR@1 ↑FID ↓Body F1 ↑Hand F1 ↑
PAMI0.8090.5470.6730.905
w/o part factorization0.7690.8200.4060.720
w/o leg and root anchors0.8060.5370.6250.882
w/o anchors0.7940.5430.4090.714

Comparison with CoDA. We retrain CoDA on the same InterAct training set as ours using the publicly released code. On GRAB test set, PAMI achieves substantially higher R@1 (0.809 vs. 0.247), and higher hand contact F1 (0.905 vs. 0.782) than CoDA. Furthermore, CoDA is optimization based method, leading to significantly slower runtime (384x slower) than our feedforward method, evaluated on A100 for generation of a sequence of 300 frames. Qualitatively, CoDA tends to always grab object using two hands. We hypothesize that it is due to the distribution of InterAct where many interactions are done with both hands. CoDA learns this distribution and its optimization does not use text information hence cannot correct such two-hand grasping tendency, yielding a much lower R1 score than ours. We will add this to appendix.

Table B. CoDA comparison on GRAB testset

MethodR@1 ↑FID ↓Body F1 ↑Hand F1 ↑Runtime/sequence
Ground Truth0.7690.0001.0001.000-
CoDA0.24712.3050.3800.78217 min
PAMI0.8090.5470.6730.9052.65 s

Dear Reviewer 67yY,

Thank you again for your detailed review. As the discussion period is coming to a close, we would be grateful to know whether our responses address your concerns, or if there is anything else we can clarify.

Best regards, The Authors

Comments on the paper

1 comment

We thank all reviewers for their time and thoughtful feedback. If our responses and additional experiments have resolved your concerns, please kindly raise your rating. Please let us know if any concerns remain, we would be happy to address them during the discussion.

We appreciate the reviewers' recognition of PAMI's state-of-the-art results (67yY) and physically plausible, semantically coherent motion aligned with text and object conditions (2vjC). They also highlighted the well-motivated representation for human–object coupling (67yY, 2vjC) and effective geometric refinement supported by convincing ablations (67yY). Reviewer JALc further recognized clear contact-quality gains from anchors even before refinement and PamiRefiner's ability to improve other generators without retraining.

On InterAct, the full PAMI framework clearly outperforms all baselines, improving hand-contact recall from 0.730 to 0.836 (+14.5%) and F1 from 0.751 to 0.821 over the previous state of the art. This gain is several times larger than the differences among the three strongest prior methods, which lie within 0.03 of each other on both metrics.

The major concerns from reviewers are limited contributions and experimental evidence for our claims. Reviewers 2vjC and JALc question the contributions of the representation and PamiRefiner, comparing our coarse stage without PamiRefiner against complete baseline pipelines. Even in this setting, our coarse stage alone already outperforms all baselines in FID, R-precision, and hand-contact metrics. PamiRefiner, however, is an integral part of our full method. It is inspired by the local geometry representations of GRAFT and GEARS, which address different tasks, and introduces new designs for refining dynamic full-body interactions. We additionally show that naively combining GRAFT and GEARS for refinement leads to much worse results (see the GRAFT + GEARS comparison in our replies to 2vjC and JALc). Our contribution is therefore not simply extending anchors from the hands to the full body, but also introducing the part-aware latent generation formulation to produce semantically accurate coarse motion and our novel refiner to generate accurate local interactions. All of these are integral parts of our method, and their contributions to performance are validated by our ablations and additional experiments.

We summarize additional experimental results below, which further demonstrate the advantages of our design:

  1. Part-based latent generation and full-body anchors (2vjC, JALc). Beyond extending the anchors, we introduce part-based latent generation. Part factorization reduces VAE reconstruction MPJPE from 38.8 to 15.2 mm and improves full-pipeline FID from 0.171 to 0.092. On the 105 test sequences with sustained non-hand contact, full-body anchors also improve body contact F1 from 0.267 to 0.302 over wrist-only anchors before refinement.

  2. PamiRefiner goes beyond combining GRAFT and GEARS (2vjC, JALc). We build a naïve GRAFT + GEARS baseline trained on the same data as PamiRefiner. This combination performs significantly worse than PamiRefiner in FID (0.117 vs. 0.092), foot skating (0.077 vs. 0.037), and penetration (0.103 vs. 0.062).

  3. Dexterous manipulation and CoDA comparison (67yY). On the GRAB subset, PAMI outperforms CoDA retrained on the same InterAct training data in hand contact F1 (0.905 vs. 0.782). Ablations also confirm the benefits of anchors and part factorization. We also clarify how our approach differs from joint-/chain-level graph interaction modeling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.