- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new method named PAMI for text-conditioned full-body human–object interaction generation. The core contributions of this paper are the proposed part-based latent generative framework and the body-part-based object voting representation. The proposed representation anchors object motion to body parts and lets the anchors vote for object movement via learned weights, achieved by PamiVAE. Then, the authors propose a refinement network that corrects the contact details using surface probes around the body and fingers. The authors conduct experiments on the InterAct dataset to validate the effectiveness of the proposed method.
Critical strengths
- The anchor-relative object representation seems effective. Experimental results show that removing the anchors clearly degrades contact quality even before refinement.
- The proposed refiner can be easily applied to other generators without retraining and improves their interaction quality.
Critical weaknesses
- The technical contribution is limited. As introduced in the related work section (lines 141–143), learned weights over wrist-relative and free object trajectories were already proposed in Dex2HOI. The body-part partition follows FrankenMotion, and the surface probes follow GRAFT and GEARS.
- The main extension is adding root and ankle anchors, but this brings almost no gain before refinement, as shown in Table 7 (w/o leg and root anchors).
- Most of the improvement in interaction quality comes from the refiner, while the baselines in Table 1 have no such post-processing.
- Before refinement, PAMI is close to LIGHT on hand contact recall (0.758 vs. 0.730). The claimed 14.5% gain only appears after refinement.
- When the same refiner is applied to LIGHT, the gap becomes much smaller, as shown in Table 9.
- The authors should discuss why PAMI has the worst foot skating ratio in Table 1.
- The evaluation protocol is unclear.
- The same baselines have different results in Table 1 and Table 9. For example, the FSR of LIGHT is 0.050 in Table 1 but 0.105 in Table 9.
- The paper claims to follow LIGHT, but uses a different R-Precision batch size and obtains different ground-truth values.
Questions for authors
See Weaknesses.
3 comments
2. Contribution of the refiner and fairness of baseline comparisons
Interaction improvement comes from the refiner while baselines don’t have that. We want to note that all baselines have their own geometry-improvement mechanisms: HOI-Diff (affordance-guided correction), CHOIS (geometric guidance), InterDiff (kinematics-informed iterative correction), and LIGHT (contact-aware guidance). It is hence unfair to compare part of our method (table 7) with the full method of baselines (table 1). Without PamiRefiner, our recall improvement over LIGHT (0.730->0.758) is still significantly better than LIGHT vs. the previous best InterDiff (0.718->0.730). This clearly shows the contribution of our part-anchored latent space formulation.
LIGHT + the same refiner has smaller gap to PAMI. Applying the frozen PamiRefiner to LIGHT leads to improvement of hand-contact F1 from 0.704 to 0.755 and full-body F1 from 0.328 to 0.408 (Table 9). Yet our complete PAMI remains significantly better than LIGHT + PamiRefiner on both measures (0.821 for hands and 0.457 for full body). LIGHT benefits from our PamiRefiner demonstrates the generalization capability of our method yet our integral design of hierarchical generation still has superior performance gain.
Foot skating is worse. We thank the reviewer for pointing this problem and revisit our evaluation code. We find there is one inconsistency in the definition of ground plane which affects FSR significantly. LIGHT defines ground plane as the lowest point of human and object meshes while our previous evaluation used the lowest point of body joints. After aligning the protocol with LIGHT, our FSR improved from 0.078 to 0.037, which is better than LIGHT (0.050) and close to the best Text2HOI (0.035). The evaluation of baselines and other metrics are unaffected. We will update this number to table 1.
3. Evaluation protocol and metric discrepancies
Why Table 1 and Table 9 differ. LIGHT outputs 3D body joint positions with separately modeled hand rotations; InterAct uses body-surface-marker trajectories. Neither directly provides the SMPL pose parameters θ needed by PamiRefiner, which predicts pose parameter updates Δθ on body and object poses. We use inverse kinematics (IK) to convert outputs of LIGHT and InterAct before applying the frozen refiner, as stated in the original submission (Lines 470–473). Table 1 evaluates original baseline outputs while Table 9 reports the baseline results after IK. IK can change measured metrics, including LIGHT FSR (0.050 in Table 1 vs. 0.105 in Table 9). Note that the FSR evaluation problem above does not affect this as the evaluation uses LIGHT’s own code. Each within-Table-9 comparison uses the same conversion, ensuring a consistent before/after comparison. We will explain this further in table 9.
LIGHT evaluation protocol. The publicly released evaluation code of LIGHT contains one problem where all contact metrics consider only the final batch instead of the full test set. We contacted the authors and they rerun the evaluation with their saved baseline results again but using a different batch size of 64. This is different from their original paper where 256 was used. We simply copied the new table they sent us and evaluate our method following their new protocol. Since batch size affects R-precision, FID, and contact metrics, penetration and FSR are recalculated using corrected code, hence the GT values we reported differ from the public version. The authors promised that they will update the new table to arXiv and we will also clarify this better in Sec. 4.
Dear Reviewer JALc,
Thank you again for your detailed review. As the discussion period is coming to a close, we would be grateful to know whether our responses address your concerns, or if there is anything else we can clarify.
Best regards, The Authors
1. Technical contribution
Technical contribution is limited. Our contributions are two fold: 1). a part-anchored generative framework that uses anchor-relative object representation to learn a compact latent space for interaction and 2). A hierarchical generation framework where PamiGen first generate coarse motion in latent space and then PamiRefiner further enhances geometric correctness in explicit space. Our method is inspired by Dex2HOI, FrankenMotion, GRAFT/GEARS but differs significantly from them.
1). Distinction from Dex2HOI and FrankenMotion: Our part-relative representation is inspired by Dex2HOI, but extends wrist-based anchoring to full-body anchors. We adopt the body-part partition from FrankenMotion and design a part-anchored latent space for coarse human-object interaction generation. In contrast, both Dex2HOI and FrankenMotion generate motion in explicit geometric space. Table 7 shows that our part factorization improves R@1 and FID before refinement. Specifically, our part-factored VAE reduces reconstruction MPJPE from 38.8 to 15.2 mm (Table 6). In the full pipeline, part factorization improves FID from 0.171 to 0.092 and full-body contact F1 from 0.327 to 0.457 (Table 2). Our PamiRefiner is another novel component that differs from existing methods and further improves interaction quality, as discussed below.
2). Distinction to GEARS/GRAFT: Our PamiRefiner draws inspiration from the geometric sensing representation from GRAFT and GEARS yet we need to address the challenges introduced in text to HOI generation. GRAFT is designed for single-frame human-scene refinement where the scene is static while PAMI needs to deal with temporally dynamic human-object interaction. GEARS assumes high quality hand and object trajectory from mocap or multi-view captures and generates the finger poses while our PamiRefiner focuses on improving finger contacts from imperfect hand-object trajectories. Naively combining GRAFT and GEARS leads to large motion jitters and finger penetrations as GRAFT is not temporal-aware and GEARS is not robust to noisy wrist trajectories.
We train GRAFT and GEARS on the same InterAct training set following the same network and training recipe. To compare with PamiRefiner, we first run GRAFT on the output of PamiGen to refine body and wrists and then run GEARS to replace fingers. The comparison is reported in table below. We can see that simple combination of GRAFT and GEARS leads to worse FID and FSR due to motion jitter, worse penetration and hand contact due to inaccurate finger generation. Our PamiRefiner harmonizes the hybrid geometry sensing probes in a new spatiotemporal transformer network and is trained with our proposed two-stream data together with additional geometric loss designs, leading to better performance across all metrics. We will add this ablation to table 3.
Root and ankle anchors brings no gain before refinement. This is due to the heavy imbalance in test set: only 105 of 907 test sequences contain non-hand contact in at least 10% of ground-truth frames, where the body anchor actually plays a role. Hence average over full test set measures mainly the effectiveness of hand anchors. We hence evaluate the contact scores of the body parts corresponding to our additional anchors on these 105 sequences, see table below. Our F1 improves from 0.267 to 0.302, clearly demonstrating the effectiveness of our method. We would like to emphasize that our contribution is not simply “extending the anchors”. Our method also contributes the part latent generation and PamiRefiner as discussed above.