- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UniHOI for joint generation of hand object interaction videos and 3D motion. The main motivation is that video and motion representations have different spatial and temporal resolutions, which makes direct interaction between them difficult. The authors introduce a unified latent space with visual anchored and motion refined states. They also use geometry supervision, adaptive spatial attention, and rhythm alignment to improve structural and temporal consistency. The method is evaluated on the TACO dataset for video generation, pose controlled video generation, and motion generation. The reported results are better than the selected baselines on most metrics, and the ablation study shows that the proposed modules are useful.
Critical strengths
-
The mismatch between video and motion representations is a reasonable problem to study. The unified latent space provides a clear way to exchange visual and 3D structural information between the two modalities.
-
UniHOI performs well on video generation, pose controlled video generation, and motion generation. The ablation study also shows that the proposed components contribute to the final performance.
Critical weaknesses
-
All experiments use TACO, and the unseen tool setting is still an internal split. This limits the evidence for broader generalization. In addition, some video baselines use released pretrained checkpoints while UniHOI is trained on the target setting, making the comparison less controlled.
-
The reported 43.1% and 22.9% reductions in “average relative motion error” are not clearly defined or directly reproducible from Table 2. Since these numbers are emphasized as major results, the calculation should be explicitly explained and verified.
-
The statistical reliability of the improvements is unclear. The paper does not report multiple runs, standard deviations, or confidence intervals. Some gains are very small, so it is difficult to determine whether they are stable and meaningful.
-
Removing ULS reduces performance, but this does not directly prove that the gain comes from resolving the proposed representation mismatch. A matched direct cross modal baseline would provide stronger evidence. The manuscript also contains several presentation issues, including an unresolved figure reference, inconsistent baseline naming, and several typographical and grammatical errors.
Questions for authors
-
How are the reported 43.1% and 22.9% reductions in “average relative motion error” calculated? Please provide the exact formula and indicate which metrics in Table 2 are included in this average, since these percentages are not directly reproducible from the reported numbers.
-
Which baselines were trained or fine tuned on TACO under the same setting as UniHOI, and which were evaluated using released checkpoints? Please clarify how training data, resolution, and optimization settings were controlled across methods.
-
What exactly is used in the “w/o ULS” ablation? In particular, is the unified latent space replaced by direct cross modal interaction with comparable model capacity, supervision, and training cost?
-
Are the reported results based on a single training run or multiple runs? If multiple runs were performed, please report variance or confidence intervals, especially for metrics where the improvement over the strongest baseline is small.
5 comments
Responses to Reviewer QRyb’s Questions
Response to Question 1: Correction and Clarification of Motion Error Reductions.
We thank the reviewer for this question. Please see our response to Weakness 2.
Response to Question 2: Clarification of Baseline Training Settings.
We thank the reviewer for this question. Due to computational constraints, we evaluate Wan2.1-14B and HunyuanVideo-13B using their publicly released pretrained checkpoints. All other baselines are fine-tuned on TACO. Detailed training configurations and implementation details are provided in supplementary Sec. A.6.3.
Response to Question 3: Analysis of Cross-Modal Direct Interaction Mechanisms.
We thank the reviewer for this question. As detailed in our response to Weakness 4, we clarify the experimental setup and provide quantitative ablations, qualitative attention-distribution analyses, and computational efficiency comparisons. Further details are provided in Sec. 4.5 of the main paper and Supplementary Secs. A.2 and A.7.
Response to Question 4: Statistical Reliability and Generation Robustness.
We thank the reviewer for this question. The originally reported results were based on a single training run. Please see our Response to Weakness 3.
Follow-up on Our Rebuttal
Thank you for your time and valuable feedback! As the author–reviewer discussion period will end in about one days, we kindly ask you to review our rebuttal and let us know whether it addresses your concerns.
Responses to Reviewer QRyb’s Questions
We sincerely thank the reviewer for recognizing the motivation, the effectiveness of our unified latent space design, and the strong generation performance. We will address your concerns point by point below.
We thank the reviewer for this suggestion. Please refer to our response to Weakness 1 raised by Reviewer qMT8 for details.
We thank the reviewer for raising this concern.
Baseline Settings. Only Wan2.1-I2V-14B and HunyuanVideo-I2V-13B use pubilc checkpoints due to computational constraints; all other baselines are fine-tuned on TACO.
Controlled Shared-Backbone Comparisons. UniHOI, EchoMotion, and Wan2.2-5B share the same pretrained video backbone, providing more direct evidence of the benefits of joint generation and fine-grained cross-modal modeling.
We thank the reviewer for pointing out this inconsistency. We identified errors in the originally reported percentages and have corrected them in the revised manuscript. Specifically, the average relative motion error reduction is calculated as the arithmetic mean of the relative error reductions across all six hand and object motion metrics (MPJPE, PA-MPJPE, MWTE, and MRE). Based on the results in Tab 3, UniHOI achieves average relative error reductions of 48.8% and 17.6% compared with LatentHOI and EchoMotion, respectively. We have clarified the calculation procedure in Supplementary Sec. A.6.1.
We thank the reviewer for this suggestion. Our additional evaluations demonstrate that UniHOI achieves more pronounced improvements in visual quality of interaction region while maintaining stable generation performance across independent runs.
The conventional video evaluation metrics used in original evaluation primarily assess global generation quality and may not fully capture improvements in visual quality within interaction regions introduced by 3D structural priors. To better assess interaction-region visual quality, we additionally introduce CLIPhand. As shown in Tab. 2 of the revised paper, UniHOI achieves 3.4% and 2.9% relative improvements over strong baselines built upon pretrained video foundation models in TI2V and pose-controlled video generation, respectively.
As shown in Supplementary Tab. 10, the multi-seed training experiments demonstrate that UniHOI maintains competitive content consistency and leading motion dynamics in video generation, while exhibiting low variability across independent runs. Detailed statistical results are provided in Supplementary Sec. A.9.
We thank the reviewer for this valuable suggestion. Our extended ablations and attention analyses provide further evidence that ULS-based interaction more effectively coordinates heterogeneous video and motion representations than direct cross-modal interaction, leading to fine-grined cross-modal correspondences and improved co-generation quality.
Clarification of w/o ULS. We clarify that the original w/o ULS variant replaces the unified latent space with direct bidirectional cross-attention. We rename this variant Direct Cross-Attn in the revised paper for clarity.
Ablation and Efficiency Analysis of Direct Interaction Mechanisms. We compare UniHOI with two direct cross-modal interaction baselines, Direct Cross-Attn and Joint Self-Attn. As shown in Tab. 4 and Fig. 14, UniHOI achieves superior video-motion generation quality with compareable computational cost.
Attention Distribution Analysis. ULS mitigates heterogeneous representation mismatches by mediating cross-modal interation through a shared latent space. As shown in Fig. 5 and Fig. 6, direct cross-modal interaction produces dispersed and unstable attention distributions, whereas UniHOI dynamically focuses on interaction regions, establishing more accurate spatiotemporal correspondences.
We thank the reviewer for carefully identifying these presentation issues. We have corrected the unresolved figure reference, standardized baseline naming, and fixed the typographical and grammatical errors throughout the revised manuscript.