- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses text-conditioned generation of hand–object interactions with rigid and articulated objects. It represents each frame with lightweight tokens and a diffusion transformer alternates attention across tokens within each frame and across frames within each token stream.
Critical strengths
Compact and structured representations for motion diffusion: The paper introduces a compact representation for bimanual hand–object motion generation by separating object states, object-centered hand positions, and low-dimensional hand-pose latents. This avoids directly diffusing high-dimensional MANO poses and provides a structured way to model the different components of the interaction sequence.
Critical weaknesses
-
The pipeline has several components that have precedents in HOI generation: ContactVAE resembles Text2HOI's contact-map VAE, ManiVAE resembles LatentHOI's GraspVAE, and ProxiRefine builds on existing contact-based pose refinement. The paper names all these components giving the impression that these are standalone contributions but these have limited novelty and perhaps the strength is the light-weight representations in the pipeline. These components (ContactVAE, ManiVAE and ArtPlanner) should be compared against the baselines where possible, not just against ground truth. And the paper should focus more on comparing network sizes or inference times to support the light-weight aspect, as the novelty is limited.
-
Limited diversity from design choices: The pipeline is largely deterministic - ArtPlanner always decodes its prior mean, DualFormer uses deterministic DDIM (noise adds a little variation), and ProxiRefine is deterministic. As a conseqeunce, DuetHOI has the lowest SD in the primary Bim-Art. setting and the lowest SD/OD ratio in 5 of 6 modes (from Tables 1 and 5). It could be that the method is seeking a mode instead of learning a good distribution. FID, prevision and recall over multiple generates trajectories would be necessary to understand the behavior.
-
The reported gains in contact and hand accuracy are not consistently accompanied by better object motion reconstruction and this is evident in the GRAB ORE results. I would like to see some analysis/explanation specifically for this metric and why its an outlier compared to baselines for GRAB.
-
I think DiffH2O from SIGGRAPH Asia 2024 is also a very relevant baseline that should be included. Further, the qualitative results should include visualizations from all baselines.
Questions for authors
-
Adding DiffH2O would be nice but even without that, model size and efficiency could be quantified against baselines
-
How can you dissect weakness #2 to see if the generations are merely sticking to ground truth trajectories and if repeated generations are yielding simialr trajectories?
Clarifications:
- It would help to understand GRAB ORE numbers with some explanation/visualizations. I want to understand how Phy is the best when ORE is the worst.