acceptodds
Under review as a conference paper at ICLR 2027

UniHOI: Video and Motion Co-Generation via a Unified Latent Space for Hand–Object Interaction

Abstract

Hand-object interaction (HOI) generation aims to synthesize physically plausible videos and 3D motion. Motion generation models capture structured 3D interaction dynamics but are limited by scarce high-quality data, while video foundation models learn rich visual priors yet lack explicit modeling of 3D geometric structure. To leverage these complementary strengths, recent works have explored video and motion co-generation frameworks that jointly model visual priors and 3D structural constraints. They often employ direct attention-based interactions to capture cross-modal dependencies between video and motion representations. However, these methods overlook the spatiotemporal granularity mismatch between heterogeneous representations, hindering fine-grained cross-modal correspondence and degrading native representations. Therefore, our key insight is to introduce a modality-shared yet interaction-comprehensive latent representation for cross-modal interaction. Motivated by this, we propose UniHOI, a unified video and motion co-generation framework that preserves modality-specific generation while constructing a unified latent space to abstract away modality-specific organization and retain complementary interaction information. Specifically, we organize this space along two complementary dimensions, visual dynamics and 3D structure, thereby establishing fine-grained spatiotemporal correspondence and enabling collaborative modeling of rich visual priors and structured 3D information. Furthermore, we introduce cross-modal rhythm alignment to promote consistent temporal dynamics between video and motion generation. Extensive experiments validate our superiority over state-of-the-art.

Primary area:
generative models
Keywords:
Hand–Object Interaction Generation, Video–Motion Co-Generation, Multimodal Diffusion Transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QRybOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniHOI for joint generation of hand object interaction videos and 3D motion. The main motivation is that video and motion representations have different spatial and temporal resolutions, which makes direct interaction between them difficult. The authors introduce a unified latent space with visual anchored and motion refined states. They also use geometry supervision, adaptive spatial attention, and rhythm alignment to improve structural and temporal consistency. The method is evaluated on the TACO dataset for video generation, pose controlled video generation, and motion generation. The reported results are better than the selected baselines on most metrics, and the ablation study shows that the proposed modules are useful.

Critical strengths
  1. The mismatch between video and motion representations is a reasonable problem to study. The unified latent space provides a clear way to exchange visual and 3D structural information between the two modalities.

  2. UniHOI performs well on video generation, pose controlled video generation, and motion generation. The ablation study also shows that the proposed components contribute to the final performance.

Critical weaknesses
  1. All experiments use TACO, and the unseen tool setting is still an internal split. This limits the evidence for broader generalization. In addition, some video baselines use released pretrained checkpoints while UniHOI is trained on the target setting, making the comparison less controlled.

  2. The reported 43.1% and 22.9% reductions in “average relative motion error” are not clearly defined or directly reproducible from Table 2. Since these numbers are emphasized as major results, the calculation should be explicitly explained and verified.

  3. The statistical reliability of the improvements is unclear. The paper does not report multiple runs, standard deviations, or confidence intervals. Some gains are very small, so it is difficult to determine whether they are stable and meaningful.

  4. Removing ULS reduces performance, but this does not directly prove that the gain comes from resolving the proposed representation mismatch. A matched direct cross modal baseline would provide stronger evidence. The manuscript also contains several presentation issues, including an unresolved figure reference, inconsistent baseline naming, and several typographical and grammatical errors.

Questions for authors
  1. How are the reported 43.1% and 22.9% reductions in “average relative motion error” calculated? Please provide the exact formula and indicate which metrics in Table 2 are included in this average, since these percentages are not directly reproducible from the reported numbers.

  2. Which baselines were trained or fine tuned on TACO under the same setting as UniHOI, and which were evaluated using released checkpoints? Please clarify how training data, resolution, and optimization settings were controlled across methods.

  3. What exactly is used in the “w/o ULS” ablation? In particular, is the unified latent space replaced by direct cross modal interaction with comparable model capacity, supervision, and training cost?

  4. Are the reported results based on a single training run or multiple runs? If multiple runs were performed, please report variance or confidence intervals, especially for metrics where the improvement over the strongest baseline is small.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer QRyb’s Questions

We sincerely thank the reviewer for recognizing the motivation, the effectiveness of our unified latent space design, and the strong generation performance. We will address your concerns point by point below.

Response to Weakness 1-1: Cross-Dataset Generalization.

We thank the reviewer for this suggestion. Please refer to our response to Weakness 1 raised by Reviewer qMT8 for details.

Response to Weakness 1-2: Controlled Baseline Comparisons.

We thank the reviewer for raising this concern.

  • Baseline Settings. Only Wan2.1-I2V-14B and HunyuanVideo-I2V-13B use pubilc checkpoints due to computational constraints; all other baselines are fine-tuned on TACO.

  • Controlled Shared-Backbone Comparisons. UniHOI, EchoMotion, and Wan2.2-5B share the same pretrained video backbone, providing more direct evidence of the benefits of joint generation and fine-grained cross-modal modeling.

Response to Weakness 2: Correction and Clarification of Motion Error Reductions.

We thank the reviewer for pointing out this inconsistency. We identified errors in the originally reported percentages and have corrected them in the revised manuscript. Specifically, the average relative motion error reduction is calculated as the arithmetic mean of the relative error reductions across all six hand and object motion metrics (MPJPE, PA-MPJPE, MWTE, and MRE). Based on the results in Tab 3, UniHOI achieves average relative error reductions of 48.8% and 17.6% compared with LatentHOI and EchoMotion, respectively. We have clarified the calculation procedure in Supplementary Sec. A.6.1.

Response to Weakness 3: Statistical Reliability and Generation Robustness.

We thank the reviewer for this suggestion. Our additional evaluations demonstrate that UniHOI achieves more pronounced improvements in visual quality of interaction region while maintaining stable generation performance across independent runs.

  • Substantial Improvements in Interaction-Region Visual Quality.

The conventional video evaluation metrics used in original evaluation primarily assess global generation quality and may not fully capture improvements in visual quality within interaction regions introduced by 3D structural priors. To better assess interaction-region visual quality, we additionally introduce CLIPhand\mathrm{CLIP}_{\mathrm{hand}}. As shown in Tab. 2 of the revised paper, UniHOI achieves 3.4% and 2.9% relative improvements over strong baselines built upon pretrained video foundation models in TI2V and pose-controlled video generation, respectively.

  • Robust Performance Across Multiple Training Runs.

As shown in Supplementary Tab. 10, the multi-seed training experiments demonstrate that UniHOI maintains competitive content consistency and leading motion dynamics in video generation, while exhibiting low variability across independent runs. Detailed statistical results are provided in Supplementary Sec. A.9.

Response to Weakness 4-1: Analysis of Cross-Modal Interaction Mechanisms.

We thank the reviewer for this valuable suggestion. Our extended ablations and attention analyses provide further evidence that ULS-based interaction more effectively coordinates heterogeneous video and motion representations than direct cross-modal interaction, leading to fine-grined cross-modal correspondences and improved co-generation quality.

  • Clarification of w/o ULS. We clarify that the original w/o ULS variant replaces the unified latent space with direct bidirectional cross-attention. We rename this variant Direct Cross-Attn in the revised paper for clarity.

  • Ablation and Efficiency Analysis of Direct Interaction Mechanisms. We compare UniHOI with two direct cross-modal interaction baselines, Direct Cross-Attn and Joint Self-Attn. As shown in Tab. 4 and Fig. 14, UniHOI achieves superior video-motion generation quality with compareable computational cost.

  • Attention Distribution Analysis. ULS mitigates heterogeneous representation mismatches by mediating cross-modal interation through a shared latent space. As shown in Fig. 5 and Fig. 6, direct cross-modal interaction produces dispersed and unstable attention distributions, whereas UniHOI dynamically focuses on interaction regions, establishing more accurate spatiotemporal correspondences.

Response to Weakness 4-2: Presentation Issues and Corrections.

We thank the reviewer for carefully identifying these presentation issues. We have corrected the unresolved figure reference, standardized baseline naming, and fixed the typographical and grammatical errors throughout the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer QRyb’s Questions

Response to Question 1: Correction and Clarification of Motion Error Reductions.

We thank the reviewer for this question. Please see our response to Weakness 2.

Response to Question 2: Clarification of Baseline Training Settings.

We thank the reviewer for this question. Due to computational constraints, we evaluate Wan2.1-14B and HunyuanVideo-13B using their publicly released pretrained checkpoints. All other baselines are fine-tuned on TACO. Detailed training configurations and implementation details are provided in supplementary Sec. A.6.3.

Response to Question 3: Analysis of Cross-Modal Direct Interaction Mechanisms.

We thank the reviewer for this question. As detailed in our response to Weakness 4, we clarify the experimental setup and provide quantitative ablations, qualitative attention-distribution analyses, and computational efficiency comparisons. Further details are provided in Sec. 4.5 of the main paper and Supplementary Secs. A.2 and A.7.

Response to Question 4: Statistical Reliability and Generation Robustness.

We thank the reviewer for this question. The originally reported results were based on a single training run. Please see our Response to Weakness 3.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your time and valuable feedback! As the author–reviewer discussion period will end in about one days, we kindly ask you to review our rebuttal and let us know whether it addresses your concerns.

Reviewer_SbZgOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniHOI targets the joint generation of videos and hand-object motions, which simultaneously generates future videos and the motions of two hands and two objects from a given initial image, initial 3D hand-object states and text. The method retains independent branches for video generation and motion generation, shares a latent space to exchange visual and 3D information, introduces Sparse Geometry Supervision, Adaptive Spatial Attention and Cross-modal Rhythm Alignment, and its improvements over the relative baselines of video generation, motion generation and their joint generation are verified on TACO Test-2.

Critical strengths

The the inherent spatiotemporal granularity mismatch between heterogeneous representations may further degrade spatial and temporal native representations. UniHOI exchanges information through an intermediate shared state and is anchored with 2D/3D geometric supervision respectively, which enables information interaction of heterogeneous granularities.

On TACO Test-2, all motion metrics obtained by UniHOI are better than the selected baseline methods, and multiple metrics of video generation also outperform the compared baselines, which demonstrates that UniHOI can achieve excellent results on TACO Test-2.

Critical weaknesses

It is clearly stated in the paper that the spatiotemporal granularity mismatch will lead to poor feature representation capability. The implementation in the paper compresses the video representation over time while the motion representation retains the frame-by-frame state, but setting the shared query to the frame-by-frame form cannot directly ensure that it establishes accurate frame-level correspondence from the compressed features; the rhythm constraint of CMRA is also not equivalent to verifying such correspondence. It is necessary to supplement the ablation that cancels frame-by-frame alignment, as well as the evaluation of frame-level correspondence or time offset error.

UniHOI reduces the number of tokens does not necessarily mean filtering out background information. w/o ULS changes the interaction structure at the same time, and the AGS/ASA ablation also does not isolate the effect of the compression state itself. It is necessary to compare direct attention with different numbers of queries under the same supervision and training budget. In addition, video actions such as SViMo and HandWorld have not been quantitatively compared as baselines, which cannot fully prove the advantages of UniHOI.

The task training, testing and ablation experiments are all concentrated on TACO, and Test-2 only verifies the geometry of unseen tools. This cannot illustrate whether the model can adapt to different collection environments, interaction types or datasets. The authors should supplement experiments on other datasets, and distinguish the effectiveness after retraining from the cross-dataset generalization without adaptation; otherwise, the scope of the conclusions should be narrowed.

Questions for authors
  1. Clarify how time granularity alignment is implemented and provide direct evidence. It is recommended to explain how frame-by-frame queries are associated with time-compressed video features, distinguish between the two goals of "establishing frame-level correspondence" and "constraining motion rhythm". Add an ablation experiment that cancels frame-by-frame query alignment, and report frame-level correspondence or time offset errors to verify the actual contribution of the time-side design.

  2. Under the supervision and training budget, compare direct attention, shared latent space and different numbers of queries, and quantify the attention distribution of interactive areas and backgrounds to verify whether compression does reduce background interference. It is recommended to add quantitative comparisons with joint generation methods such as SViMo and HandWorld.

  3. It is recommended to conduct verification on paired video-hand-object motion datasets other than TACO, and separately report the performance after retraining on the target dataset and the cross-dataset results without adaptation. If supplementation is not possible, the conclusion should be clearly limited to the unseen tool geometry generalization of TACO Test-2.

  4. Evaluate the fidelity with GT, text condition compliance, and interaction rationality separately, and avoid judging the quality of generation solely by PSNR/SSIM or the motion error of a single GT. At the same time, add indicators such as trajectory correspondence, speed consistency or event time offset between the generated video and 3D motion to directly verify cross-modal synchronization.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer SbZg’s Concerns

We sincerely thank the reviewer for recognizing the effectiveness of our unified latent space design in addressing spatiotemporal granularity mismatches, as well as its strong motion generation performance. We will address your concerns point by point below.

Response to Weakness 1: Fine-Grained Cross-Modal Temporal Correspondence

We thank the reviewer for this suggestion. We clarify that frame-level queries provide frame-specific representation slots but do not alone guarantee fine-grained temporal correspondence. UniHOI progressively achieve it through two mechanisms:

  • Cross-Modal 3D RoPE Alignment: Cross-Modal 3D RoPE (Sec. A.1) provides consistent temporal positional encoding to establish coarse-grained temporal associations, guiding frame-level latent states to extract relevant visual interaction cues from compressed video representations.

  • Geometry-Guided Temporal Discrimination: AGS employs sparse geometric supervision to encourage latent states to distinguish frame-wise interaction dynamics and establish fine-grained temporal correspondences.

We agree that CMRA builds upon established frame-level correspondences to further align motion dynamics across modalities. Our additional analyses provide further evidence that UniHOI establishes fine-grained frame-level temporal correspondences through the unified latent space. Detailed analyses are provided in Supplementary Sec. A.2.1.

  • Quantitative Ablations. As shown in Tab. 5, we evaluate the contributions of frame-level state organization, cross-modal 3D RoPE, AGS, and CMRA, and quantitatively assess frame-level video-motion correspondence.

  • Spatiotemporal Attention Analysis. As shown in Fig. 6 and Fig. 7, UniHOI exhibits temporally concentrated attention patterns and frame-specific spatial attention dynamically focused on interaction regions, demonstrating its ability to capture fine-grained interaction dynamics and establish accurate frame-level cross-modal correspondences.

Response to Weakness 2-1: Disentangling Query Reduction and Spatial Selectivity

We thank the reviewer for this suggestion. We clarify that UniHOI does not rely on query reduction to suppress background interference. Instead, AGS and ASA jointly guide cross-modal attention focus on crucial interaction regions and suppress irrelevant background responses. Detailed analyses are provided in Supplementary Sec. A.2.2.

  • Effectiveness of ULS-Based Interaction. Compared with direct cross-modal attention, ULS mitigates the spatiotemporal granularity mismatch between heterogeneous video and motion representations through a unified latent space. As shown in Tab. 4, UniHOI achieves superior overall video-motion co-generation performance over direct interaction baselines.

  • Query capacity does not directly determine spatial selectivity. Reducing the number of queries alone does not explicitly prioritize interaction-relevant information. As shown in Tab. 6 and Fig. 8, query reduction alone does not reduce background attention density or yield consistent improvements in generation quality.

  • AGS and ASA enable interaction-aware spatial integration. AGS guides shared states focus on interaction-relevant visual cues, while ASA selectively integrates 3D structural information into interaction visial regions. Together, they concentrate bidirectional cross-modal attention on interaction regions, suppress background interference, and facilitate fine-grained spatiotemporal correspondences.

Response to Weakness 2-2: Comparison with hand-object Joint Generation Methods

  • SViMo employs synchronized diffusion with 2D-projected motion representations and an additional 3D motion generation module for closed-loop feedback. In contrast, UniHOI jointly models native 3D motion and video representations, enabling fine-grained bidirectional cross-modal interaction through the unified latent space. SViMo's implementation and pretrained checkpoints are not publicly available, preventing a reproducible quantitative comparison.

  • HandWorld integrates cross-modal information primarily through shared conditioning network, without continuous bidirectional interaction between video and motion representations during generation. UniHOI enables progressive bidirectional cross-modal interaction through a unified latent space, promoting fine-grained cross-modal consistency. We have further included comparisons with HandWorld in Tab. 1 and Tab. 2 of the revised main paper. The results demonstrate UniHOI's superior performance in video–motion consistency.

Response to Weakness 3: Cross-Dataset Generalization Experiments

We thank the reviewer for this suggestion. Please refer to our response to Weakness 1 raised by Reviewer qMT8 for details.

AuthorsOpenReview ↗

Responses to Reviewer SbZg’s Questions

Response to Question 1: Fine-Grained Cross-Modal Temporal Correspondence

We thank the reviewer for this suggestion. As detailed in our response to Weakness 1, we clarify the implementation details of frame-level temporal alignment and provide additional ablation studies and spatiotemporal attention analyses to validate the proposed alignment mechanisms in Supplementary Sec. A.2.1.

Response to Question 2: Disentangling Query Reduction and Spatial Selectivity

We thank the reviewer for this suggestion. As detailed in our response to Weakness 2, we provide additional ablation studies comparing direct attention, ULS-based interaction, and different query counts, together with quantitative analyses of attention distributions over interaction and background regions in Supplementary Sec. A.2.2.

Response to Question 3: Cross-Dataset Generalization Experiments.

We thank the reviewer for this question. Please see our response to Weakness 3.

Response to Question 4: Evaluation of Cross-Modal Alignment

We thank the reviewer for this suggestion! Our original evaluation already covers GT-based fidelity (PSNR/SSIM and motion errors) and interaction plausibility (IV, ID, and CRE). We further introduce video–text consistency and frame-level video–motion consistency metrics in Tab. 1 of the revised main paper. UniHOI consistently outperforms SOTA across these metrics, demonstrating its effectiveness in establishing fine-grained cross-modal consistency.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your time and valuable feedback! As the author–reviewer discussion period will end in about one days, we kindly ask you to review our rebuttal and let us know whether it addresses your concerns.

Reviewer_qMT8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a unified framework for jointly generating hand–object interaction (HOI) videos and 3D motion. Existing motion generation methods capture rich structural information but are limited by the scarcity of high-quality training data. In contrast, video generation methods leverage pretrained video diffusion models to generate high-quality HOI videos but lack explicit modeling of 3D structure.

Existing co-generation methods use direct cross-modal attention to learn complementary dependencies between video and motion representations. However, they overlook the inherent spatiotemporal granularity mismatch between these heterogeneous representations, leading to interference from spatially redundant information.

To address this, the paper introduces a modality-shared unified latent space that facilitates cross-modal interactions while preserving complementary visual and 3D structural information.

Critical strengths
  • Well-motivated approach: The paper identifies the spatiotemporal granularity mismatch between video and 3D motion representations. The proposed unified latent space is a reasonable approach to addressing this problem and demonstrates promising results, particularly for motion generation.

  • Clear presentation: The paper is well-written, well-structured, and easy to follow.

Critical weaknesses
  • Limited Generalization: All reported experiments are conducted on the TACO dataset. Although the authors evaluate generalization to unseen tool geometries within TACO, there are no cross-dataset experiments to establish the generalization ability of the proposed approach.

  • Limited Evaluation of Cross-Modal Alignment: The ablation studies suggest that the individual components contribute positively to the overall performance. However, there are no direct experiments evaluating how effectively the proposed unified latent space aligns video and 3D motion representations.

  • Uneven Improvements Across Modalities: From Tables 1 and 2, the improvements in video generation appear relatively marginal, whereas the improvements in motion generation are substantial. This raises questions about how effectively the unified architecture facilitates the transfer of complementary information between the two modalities. Ideally, one would expect more substantial improvements in both tasks if cross-modal knowledge transfer is effective.

Questions for authors
  • Qualitative Results: Since the video generation branch is based on an image-to-video diffusion model, could the authors clarify why the different methods in Fig. 3 produce relatively similar outputs? It would be helpful to understand the role of the initial image and other conditioning signals in producing these results.

  • Baseline Training: Could the authors clarify whether all baselines were retrained for this new task?

  • Cross-Modal Alignment: The authors should consider including a direct quantitative evaluation of the alignment between generated videos and their corresponding 3D motions, as this is one of the central motivations of the proposed framework.

  • Generalization: Additional experiments on datasets beyond TACO would help demonstrate the robustness and generalization of the proposed approach.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer qMT8’s Concerns

We sincerely thank the reviewer for recognizing the strong motivation, clear presentation, and promising results of our work. We will address your concerns point by point below.

Response to Weakness 1: Cross-Dataset Generalization Experiments.

We thank the reviewer for this suggestion! Additional experiments on OakInk2 dataset demonstrate UniHOI's robust zero-shot generalization in 3D motion and contact modeling, as well as its consistent effectiveness across domain distributions, as detailed in Supplementary Sec. A.8.

  • Zero-Shot Cross-Dataset Generalization:

We directly evaluate TACO-trained models on the OakInk2 test set. As shown in Supplementary Tab. 9, UniHOI achieves the best motion accuracy, contact consistency and video–motion consistency compared with SOTA. The results demonstrate that UniHOI facilitates fine-grained cross-modal alignment through the unified latent space, enabling the motion generation branch to effectively leverage rich visual priors.

  • Effectiveness Across Different Datasets:

We further retrain UniHOI and relevant baselines on OakInk2. As shown in Supplementary Tab. 7 and Tab. 8, UniHOI achieves the best overall performance in all metrcis, demonstrating its effectiveness across different data distributions.

Response to Weakness 2: Evaluation of Cross-Modal Alignment.

We thank the reviewer for this suggestion! We further introduce video–text consistency and frame-level video–motion consistency metrics in Tab. 1 of the revised main paper. UniHOI consistently outperforms SOTA across these metrics, demonstrating its effectiveness in establishing fine-grained cross-modal consistency.

Response to Weakness 3: Effectiveness of Cross-Modal Knowledge Transfer.

We thank the reviewer for this insightful observation. Our additional evaluation reveals more pronounced improvements in interaction-region visual quality, further supporting UniHOI's effectiveness of complementary information transfer between the two modalities.

The conventional video evaluation metrics used in original evaluation primarily assess global generation quality and may not fully capture improvements in visual quality within interaction regions introduced by 3D structural priors. To better assess interaction-region visual quality, we additionally introduce CLIPhand\mathrm{CLIP}_{\mathrm{hand}}. As shown in Tab. 2 of the revised paper, UniHOI achieves 3.4% and 2.9% relative improvements in TI2V and pose-controlled video generation.

AuthorsOpenReview ↗

Responses to Reviewer qMT8’s Questions

Response to Question 1: Clarification of Visual Similarity in Fig. 3.

We thank the reviewer for this careful observation. Since all methods are conditioned on the same initial frame and text prompt to ensure a fair comparison, Fig. 3 shows similarities in scene layouts and initial interaction states across different methods.

However, existing 2D video generation methods often suffer from severe structural distortions and unstable contact relationships in Fig. 3. By effectively incorporating explicit 3D structure into the video generation branch, UniHOI maintains more consistent hand-object geometry and physically plausible interactions.

Response to Question 2: Clarification of Baseline Training Settings.

We thank the reviewer for this question. Due to computational constraints, we evaluate Wan2.1-14B, HunyuanVideo-13B using their public pretrained checkpoints. All other baselines are fine-tuned on TACO training split. Detailed training configurations and implementation details are provided in supplementary Sec. A.6.3.

Response to Question 3: Evaluation of Cross-Modal Alignment.

We thank the reviewer for this suggestion. Please refer to our response to Weakness 2 for details.

Response to Question 4: Cross-Dataset Generalization Experiments.

We thank the reviewer for this suggestion. Please refer to our response to Weakness 1 for details.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your time and valuable feedback! As the author–reviewer discussion period will end in about one days, we kindly ask you to review our rebuttal and let us know whether it addresses your concerns.

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback. In their assessments of the paper's strengths, all three reviewers (qMT8, SbZg, and QRyb) recognized our solid motivation, sound architecture, and promising performance. In response to their suggestions, we made the following substantial revisions during the rebuttal.

1. Experimental Enhancements

  • Zero-Shot Generalization (qMT8, SbZg, and QRyb) and Statistical Reliability (QRyb): We added extensive comparisons on OakInk2 dataset, including target-domain retraining and zero-shot cross-dataset evaluation (Appendix A.8), along with multi-seed experiments (Appendix A.9). Experiments confirm UniHOI's strong generalization capability and stable generation performance.
  • Comprehensive Ablation Analysis (SbZg and QRyb): We substantially expanded our ablation studies, including comparisons between the Unified Latent Space (ULS) and direct cross-modal interaction (Sec. 4.6 and Appendix A.2), cross-modal temporal alignment (Appendix A.2.1), and spatial selectivity (Appendix A.2.2) within ULS. Attention analyses further show that ULS achieves more temporally aligned and spatially focused attention patterns, enabling fine-grained cross-modal correspondences.
  • Controlled Video Foundation Model Comparisons (QRyb): We expanded our baseline comparisons to include LoRA-fine-tuned Wan2.2-14B and HunyuanVideo-13B, with evaluations at both full-frame and hand-region levels (Appendix A.4.2). UniHOI outperforms both large-scale video foundation models across all hand-region metrics, demonstrating substantial improvements in fine-grained HOI video generation.
  • Comprehensive Video–Motion Co-Generation Evaluation (qMT8 and SbZg) : We introduced quantitative video–motion consistency metrics (Table 1) and included an additional video–motion co-generation baseline (HandWorld). These experiments demonstrate that ULS enhances cross-modal co-generation through effective collaborative modeling of visual and 3D motion information.

2. Further Clarifications and Revisions

  • Spatiotemporal Alignment within ULS: We further clarified how ULS establishes fine-grained spatiotemporal correspondences between heterogeneous video and motion representations through temporal alignment (Appendix A.1) and spatial selectivity.
  • Experimental Details and Corrections: We clarified training and evaluation settings, corrected the average relative motion error reduction, and resolved inconsistencies in terminology and formatting to improve reproducibility.

We are encouraged by the reviewers' positive feedback. Reviewer SbZg acknowledged that our revisions addressed their concerns and raised their score. Reviewer QRyb also recognized that our revisions resolved their primary concerns and increased their score, while expressing remaining reservations about experimental fairness. In response, we further strengthened our baseline comparisons through fairer and more comprehensive evaluations, demonstrating substantial improvements in fine-grained HOI video generation. We believe these additional experiments provide further evidence addressing the remaining concerns regarding experimental fairness and video generation quality. We sincerely thank the Area Chair and all reviewers for their time and consideration.

Sincerely, The Authors of ICLR 2027 Submission 57534

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.