acceptodds
Under review as a conference paper at ICLR 2027

FastInteract: Few-Step Human-Object Interaction Video Generation via Attention Perturbation

Abstract

Few-step distillation accelerates video diffusion models, but preserving coherent human–object interaction (HOI) while maintaining reference product consistency remains challenging. We introduce **A**ttention **P**erturbation (**AP**) to provide interaction guidance for distribution matching distillation (DMD). AP compares teacher predictions by selectively enabling or blocking target-video queries' access to conditioning keys and values, with inputs and all other attention connections fixed. An auxiliary HOI representation predicted by the student supplies an interaction-enhancement response, and a second contrast using product-reference attention provides a preservation direction. We coordinate these responses through a one-sided projection that removes the component of the enhancement response opposing the preservation direction before augmenting the DMD target. We further curate a multimodal HOI video dataset **_InteractVerse_** from product-oriented livestreams, with aligned person and product references, text, audio, motion, and spatial annotations. Experiments and ablations demonstrate that our method improves interaction plausibility and reference consistency over vanilla DMD.

Primary area:
generative models
Keywords:
Human–Object Interaction, Video Generation, Distribution Matching Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BwywOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets few-step human–object interaction video generation. It makes two contributions. First, it introduces InteractVerse, a dataset of 37,477 curated clips totaling approximately 64.1 hours, with aligned person/product references, text, audio, motion context, pose sequences, and interaction-sensitive spatial annotations. Second, it proposes FastInteract, which distills a CoInteract teacher into a four-step generator using attention perturbation: one response enhances HOI structure, a second response characterizes product preservation, and a one-sided projection coordinates the two before augmenting the DMD target.

On HOIVG-Bench, the four-step student improves VLM-QA from 70.159 for vanilla DMD to 76.957, and product-reference DINO similarity from 0.614 to 0.628. The paper reports comparisons with many-step and few-step baselines, component ablations, and qualitative coefficient probes. The evidence supports a useful interaction–appearance trade-off, while a gap to the many-step teacher remains on interaction quality.

Critical strengths

The data and modeling contributions form a coherent whole. InteractVerse provides aligned appearance, semantic, temporal, and spatial supervision at a scale beyond several controllable HOI datasets (Section 3; Appendix B). The curation pipeline combines technical filtering, face/hand/product detection, reference extraction, verification, and interaction-aware captioning. This is a reusable resource rather than a collection of captioned videos. FastInteract then addresses a corresponding modeling difficulty: retaining interaction structure and product appearance when generation is compressed to four steps.

The ablations explain the interaction–appearance trade-off. In Table 3, Full-Visibility AP improves VLM-QA from 70.159 to 75.013, but lowers product DINO similarity from 0.614 to 0.602. Restricting the intervention to target-video queries raises VLM-QA to 76.026 and DINO_obj to 0.609. Adding the one-sided projection further raises them to 76.957 and 0.628. This progression is consistent with the proposed mechanism: attention perturbation supplies most of the interaction gain, while the restricted pathway and projection recover product consistency. The coefficient probes in Appendix D.2 show the same tension qualitatively.

The empirical comparison is broad enough to be informative. Among the listed few-step baselines in Table 2, the four-step student leads on seven of eight reported metrics, including VLM-QA and DINO_obj. The authors also disclose that several baselines require a single combined reference image and construct it accordingly. This interface difference makes the within-framework ablations the stronger evidence, but the comparison remains useful.

Critical weaknesses

The VLM-QA protocol and the attribution of gains need more complete documentation. Appendix D.1 describes the 50-question bank, the four categories, the category weights, the harmonic mean, and the critical-failure caps, but it does not provide the actual questions, the judging prompt, the failure thresholds, or the frame-sampling settings. Human agreement and uncertainty estimates are also missing. This matters for the smaller margins, such as the 0.931-point VLM-QA gain from projection over TQ-AP and the 1.071-point margin over LiveAvatar. The ablations support the method, but they do not separately establish how much InteractVerse contributes to the reported performance. A matched-data ablation would separate the dataset contribution from the distillation contribution.

The residual failure modes and practical operating range are not characterized in depth. The full model remains 5.111 VLM-QA points below the 40-step CoInteract teacher, so the improvement over vanilla DMD does not eliminate the loss of interaction quality during distillation. Appendix D.2 shows that stronger HOI enhancement can introduce product drift and artifacts, but this is a coefficient probe rather than a systematic account of failures of the final model. Appendix A explicitly excludes multiple people or objects and notes that real-time generation still requires further acceleration. Representative final-model failures, category-level results explaining the remaining teacher gap, and measured latency at a stated resolution and clip length would strengthen the practical assessment. Four versus 40 steps establishes a step reduction, not a measured tenfold speedup.

The cross-method comparison is not fully controlled across conditioning interfaces. LiveAvatar, FlashTalk, and LongCat-Video-Avatar 1.5 require a single image, so the authors combine the person and product references into one image. Reference-consistency metrics still use the original separate references. The authors disclose this adaptation, but it limits how much the cross-method ranking can establish about the distillation design by itself; the component ablations remain the more direct evidence.

Questions for authors

Metric reliability: Can you release the complete VLM-QA protocol, including the question bank, judging prompt, critical-failure thresholds, and frame-sampling procedure? Can you report paired uncertainty estimates or human agreement on a representative subset, and break down the results by action, contact, motion synchronization, and structural plausibility? Such a breakdown would clarify which errors account for the gain over vanilla DMD and the remaining gap to the teacher.

Data contribution: Which comparisons keep the training data fixed? Please specify the evaluation sample size, any train/benchmark overlap checks, and how annotation quality was verified. What evidence separates gains from InteractVerse from gains from the distillation method? A matched data ablation would clarify this directly.

Failure analysis and efficiency: Which failure types persist in the full model under its supported single-person/single-object setting? Please provide representative cases and, where available, their frequency. Please also report same-hardware student/teacher latency at a stated resolution and clip length, together with the training configuration. Extending to multiple entities or real-time streaming is future work rather than a prerequisite for acceptance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bwyw

W1 / Q1 (a). HOI user study

We agree that broader evaluation coverage and validation against human judgments would strengthen the evidence. We conducted a perceptual user study with 12 evaluators. Each evaluator was shown 10 test cases randomly sampled from HOIVG-Bench. For each case, videos generated by the compared methods were presented in a blind, randomized order. Evaluators ranked the videos separately according to three criteria: Interaction Plausibility, Object Consistency, and Human/Background Consistency. We report the mean rank for each criterion, with lower values indicating better perceived quality.

User study results. Best and second-best values across all evaluated methods are bold and underlined, respectively.

MethodInteraction Plausibility ↓Object Consistency ↓Human/Background Consistency ↓
Phantom5.27505.20835.7917
VACE4.95004.41673.8167
HuMo4.71676.44176.6333
CoInteract (Teacher)4.11676.45833.6500
LiveAvatar5.69174.25005.0583
FlashTalk5.22504.35004.7583
SkyReels-V35.52505.82506.8167
LongCat-Video-Avatar 1.55.01674.2167‾\underline{4.2167}4.7667
Ours4.4833‾\underline{4.4833}3.83333.7083‾\underline{3.7083}

Even compared with multi-step models, ours outputs achieve the lowest mean rank in Object Consistency and the second-lowest mean rank in both Interaction Plausibility and Human/Background Consistency. To better understand the differences between the user-study rankings and the evaluation, we collected follow-up feedback from the evaluators.

For Interaction Plausibility, evaluators perceived the motion in our outputs more closely matched how people naturally present and interact with products, whereas some competing outputs exhibited exaggerated movements that appeared less realistic. For Object Consistency, evaluators noted oversaturated product appearances in competing outputs and preferred the overall consistency of our product appearance. For Human/Background Consistency, many evaluators noted that our outputs preserved person identity while maintaining good background consistency, faithfully reproducing the background in the reference image.

A randomly selected set of 20 qualitative cases will also be provided for inspection at User Study Page. These examples are intended to help reviewers directly assess the quality of the generated videos.

W1 / Q1 (b). VLM-QA protocol and category-level results

We have prepared the complete VLM-QA protocol for anonymous release at VLM-QA, including all 50 questions, applicability rules, judging prompts, exact failure conditions, category/overall critical-failure caps, and executable sampling and scoring code. A locally deployed Qwen3.8-27B judges each applicable question independently. Inputs include the full video requested at 16 fps, a uniformly sampled 17-frame overview including both endpoints, and four additional frame strips for contact and structure. Applicability is frozen before inspecting generated outputs and shared across methods; uncertain judgments receive zero credit.

All three methods are evaluated on the same 135 HOIVG-Bench cases. All scores are on a 0–100 scale, with higher values better. Motion refers to hand–object synchronization.

MethodOverallAction / completionContact / supportMotion synchronizationStructural plausibility
CoInteract (Teacher)82.06885.67589.00299.86597.827
Vanilla DMD70.15981.79477.04293.50595.049
FastInteract76.95782.19787.73596.87294.235

Our largest gains over vanilla DMD are in contact/support (+10.693) and motion synchronization (+3.367). The largest remaining deficits relative to the teacher are in structure (−3.593-3.593) and action completion (−3.479-3.479).

AuthorsOpenReview ↗

Response to Reviewer Bwyw

W1 / Q2. evaluation setting and Dataset contribution

Evaluation size. All reported automatic metrics are evaluated on the 135 test cases in HOIVG-Bench, released by OmniShow [1]. InteractVerse was collected from real e-commerce videos on publicly accessible livestreaming platforms, whereas HOIVG-Bench uses AI-generated person and product references. The datasets were constructed independently, and HOIVG-Bench is used only for evaluation.

Annotation quality. We apply additional VLM-based verification after image editing and mesh reconstruction. For reference extraction, the VLM checks whether the extracted person and product references preserve the identity in the source video. Samples with mismatched references, missing entities, or severe visual artifacts are discarded. We also use Qwen-Omni to filter out videos with low-quality, unclear, or noisy audio. These checks reduce the propagation of preprocessing errors into training.

Dataset contribution. The main limitation of SparseHOI-5K is its restricted identity and product coverage: its 4,850 clips contain only 34 person identities and approximately 1,000 product reference images. Generating additional captions and extracting reference images do not expand the underlying set of people, products, or recorded interactions. In contrast, InteractVerse contains 37,477 clips spanning 53 product categories and diverse interaction types, with details available on our InteractVerse Dataset Page. The clips are collected from different e-commerce livestreams at different times, providing broad coverage of person identities and products. After excluding a small number of repeated products and person identities, the dataset contains over 30,000 distinct pairs of product and person reference images. Our LiveAvatar competitor uses approximately 400,000 training samples for distillation, whereas SparseHOI-5K contains fewer than 5,000 clips. Based on this distillation experience, we consider SparseHOI-5K insufficient in both size and diversity to meet the needs of our distillation task.

W2 / Q3. Representative failure cases

We provide concrete failure cases in which the model fails to generate interactions between handheld objects and body parts other than the hands. Representative examples are available at Failure Case Page. We will incorporate these examples and a dedicated failure-case analysis into the revised appendix.

W2 / Q3. Measured efficiency and training configuration

On the same NVIDIA H20, generating a one-second video at 480 × 832 resolution takes 402 seconds for the 40-step teacher and 39.5 seconds for the four-step student, corresponding to a 10.18× measured end-to-end speedup. These timings include the full encoding and decoding pipeline. During denoising, the student eliminates the additional CFG pass, reducing the number of denoiser evaluations from 80 (40 × 2) to four, a 20× reduction.

Training was conducted on 32 NVIDIA H200 GPUs with 140 GB of device memory per GPU, using BF16 precision, Fully Sharded Data Parallel (FSDP), and a learning rate of 2×10−62 \times 10^{-6}. The model was trained for 5,000 steps over approximately three days.

W3. Conditioning-interface differences

Our task is to generate plausible interactions while preserving the person identity and product appearance specified by two separate reference images. LiveAvatar, FlashTalk, and LongCat-Video-Avatar 1.5 do not natively support this conditioning interface. We therefore use Qwen-Image-Edit to construct a composite first frame for these baselines. This adaptation may alter appearance details, and it also provides an explicit person–product spatial arrangement and initial interaction configuration.

We evaluate every method against the original person and product images, because the task's target identities are the original pair rather than the composite edit. The composite exists only because three baselines cannot ingest two separate images; it is an input workaround, not a new task specification. Scoring against it would instead measure the remaining six methods against an image they never received. Scoring each method against the input it actually received would avoid penalizing those three, but it would make the reference method-dependent: the same column would then measure fidelity to two different images. We therefore fix one task-defined target across all methods. Although the conditioning-interface asymmetry for LiveAvatar, FlashTalk, and LongCat remains, our latest user study provides complementary evidence: our evaluated outputs achieve the best mean rank in Object Consistency and a near-best mean rank in Human/Background Consistency.

References

[1] Zhou et al. OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation. ICML 2026.

Thanks for your response and the additional experiments. They address my concerns and support my initial positive assessment, so I will maintain my rating of 3

Reviewer_oK88OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets few-step human–object interaction (HOI) video generation from a prompt plus separated person and product references, starting from CoInteract, a 40-step teacher whose real-score target loses interaction fidelity under standard DMD distillation. Its mechanism, Attention Perturbation (AP), runs the teacher twice — with and without a mask blocking a chosen conditioning token set from video queries — converts the difference of the two predicted means into score space and adds it to the distillation target, so the 4-step student carries it with no inference-time overhead; a one-sided gradient-surgery projection then removes the component conflicting with a product-preservation response. It also contributes InteractVerse, 37,477 clips (~64.1 h) curated from real-world e-commerce livestreams, which Sec. 3.1 commits to releasing. On HOIVG-Bench, with VBench, DINOv2, ArcFace, SyncNet and a self-built 50-question VLM-QA metric, the student leads on seven of eight metrics among few-step methods — but scores 76.957 on VLM-QA against its own teacher's 82.068. There is no code or repository link.

Critical strengths
  • AP puts the corrective signal in the distillation target rather than at inference, so the interaction gain comes at zero deployment cost, and the ablation shows the mechanism composes rather than merely adds.

  • The authors are candid about exactly where the mechanism stops working, which raises my confidence in everything else they report.

Critical weaknesses
  • The primary quality metric is a self-built instrument with two stages of critical-failure capping and a fixed-weight harmonic mean, it is never validated against human judgment, and on it the student loses to its own teacher by 5.11 points.

  • Layers/heads never ablated. AP is defined entirely through which attention entries the mask blocks, yet the layer/head choice is never varied, never justified and (as far as I can determine) never stated as a value. Given that ViPO and STAR both show attention-selection choices materially change results, this is a gap in the mechanism's own specification.

  • Baseline convention asymmetry. LiveAvatar, FlashTalk and LongCat receive a Qwen-Image-Edit composite reference because they cannot accept separated person+product references — but DINO-id (0.4153) and FaceSim (0.789) are still computed against the original separated references. The proposed method is therefore scored against a reference representation its baselines were never given, biasing exactly the two reference-consistency metrics on which the "seven of eight" claim partly rests.

Questions for authors

Please report human agreement with VLM-QA. Even ~100 clips rated by 3 annotators, with correlation to VLM-QA and to each individual metric in Table 2, would let me judge whether the 76.957 vs 82.068 teacher gap and the "seven of eight" ranking track perceived interaction quality.

5 comments
AuthorsOpenReview ↗

Response to Reviewer oK88

W1 / Q1. Human validation of VLM-QA

We agree that our automatic metric should be validated against human judgments. To assess their agreement, we conducted a blinded user study with 12 evaluators. Each evaluator was shown 10 test cases randomly sampled from HOIVG-Bench. For each case, videos generated by the compared methods were presented in a blind, randomized order. Evaluators ranked the videos separately according to three criteria: Interaction Plausibility, Object Consistency, and Human/Background Consistency. We report the mean rank for each criterion, with lower values indicating better perceived quality.

User study results. Best and second-best values across all evaluated methods are bold and underlined, respectively.

MethodInteraction Plausibility ↓Object Consistency ↓Human/Background Consistency ↓
Phantom5.27505.20835.7917
VACE4.95004.41673.8167
HuMo4.71676.44176.6333
CoInteract (Teacher)4.11676.45833.6500
LiveAvatar5.69174.25005.0583
FlashTalk5.22504.35004.7583
SkyReels-V35.52505.82506.8167
LongCat-Video-Avatar 1.55.01674.2167‾\underline{4.2167}4.7667
Ours4.4833‾\underline{4.4833}3.83333.7083‾\underline{3.7083}

Even compared with multi-step models, ours outputs achieve the lowest mean rank in Object Consistency and the second-lowest mean rank in both Interaction Plausibility and Human/Background Consistency. To better understand the differences between the user-study rankings and the evaluation, we collected follow-up feedback from the evaluators.

For Interaction Plausibility, evaluators perceived the motion in our outputs more closely matched how people naturally present and interact with products, whereas some competing outputs exhibited exaggerated movements that appeared less realistic. For Object Consistency, evaluators noted oversaturated product appearances in competing outputs and preferred the overall consistency of our product appearance. For Human/Background Consistency, many evaluators noted that our outputs preserved person identity while maintaining good background consistency, faithfully reproducing the background in the reference image.

A randomly selected set of 20 qualitative cases will also be provided for inspection at User Study Page These examples are intended to help reviewers directly assess the quality of the generated videos.

W2. AP layer/head configuration and ablation

Both the HOI and product-preservation interventions apply to all attention layers and all heads, as specified in Appendix C.4. This configuration applies a consistent intervention to the designated attention pathways throughout the network. The HOI intervention changes target-video queries' access to HOI keys and values, while the product-preservation intervention changes their access to product-reference keys and values.

Our theoretical analysis and experimental results demonstrate the effectiveness of this intervention. The ablations in Table 3 evaluate attention visibility and the one-sided projection under this shared configuration. Layer-specific ablations require additional distillation training, which we could not complete within the time available. We plan to further investigate how different layers influence the effectiveness of AP in future work.

AuthorsOpenReview ↗

Response to Reviewer oK88

W3. Reference construction and consistency evaluation

Our task is to generate plausible interactions while preserving the person identity and product appearance specified by two separate reference images. LiveAvatar, FlashTalk, and LongCat-Video-Avatar 1.5 do not natively support this conditioning interface. We therefore use Qwen-Image-Edit with potential minor appearance variations to construct a composite first frame for these baselines, enabling single-image conditioning and providing a unified person–product spatial arrangement and initial interaction configuration.

We evaluate all methods against the original person and product images, as their identities define the target of the HOI video generation task. The composite image is introduced solely as an input adaptation for the three baselines that cannot accept two separate images; it does not redefine the task target. Evaluating against the composite image would instead assess the other methods against an image they never received. Conversely, evaluating each method against its actual input would introduce method-dependent references, making the results incomparable. We therefore use the same task-defined reference across all methods to consistently assess the preservation of the original person and product identities in the generated videos.

Although an asymmetry in conditioning interfaces remains for LiveAvatar, FlashTalk, and LongCat, our latest user study provides complementary evidence: our method achieves the best mean rank in Object Consistency and a near-best mean rank in Human/Background Consistency.

Thank you for the response. The additional results are helpful, but my concerns are only partially addressed.

  • VLM-QA: Mean ranks demonstrate human preferences, but not agreement with VLM-QA. Please report human–VLM-QA rank correlation, the number of unique clips, and inter-rater agreement. The table also appears incomplete: several values for LongCat and Ours are missing. Moreover, Ours scores 3.8333 on Human/Background Consistency, behind CoInteract (3.6500) and VACE (3.8167), so it appears third-lowest rather than second-lowest.

  • AP configuration: Applying AP to all layers and heads resolves the specification ambiguity. I understand the cost of layer-specific retraining; please acknowledge the unexplored layer sensitivity as a limitation.

  • Reference evaluation: I agree that the original references provide a reasonable common target. My remaining concern is unequal input information: three baselines are scored against original references they never directly observed, after the composite edit may have altered identity or appearance. Please report the composite images’ own DINO-id/FaceSim against the originals to quantify this loss, or explicitly acknowledge this confound. The user study is complementary but does not isolate it.

Overall, W2 is clarified; W1 and W3 still need the above quantitative clarification.

Reviewer_zBKsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FastInteract, a 4-step HOI video generator for controllable HOI video generation. This paper curates a HOI video dataset with complete HOI related annotations. Based on this dataset and existing HOI video generator, this paper utilizes distribution matching distillation(DMD) with attention perturbation(AP) to learn a few-step HOI video generator. AP will perturb attention from target-video queries to HOI conditioning keys and values derived from the student’s prediction to augment the HOI quality and to preserve the reference object appearance. The AP can be combined with classifier-free guidance(CFG) to learn a better HOI generation student. Quantitative and qualitative results indicate the strong performance of the proposed framework compared to existing works. The effectiveness of the proposed AP is also well ablated.

Critical strengths
  1. The paper contributes an HOI video dataset with complete HOI annotations and a relatively large number of real-world e-commerce videos compared to existing personalized HOI video dataset, which would benefit the development of this field. The model finetuned on this curated dataset produce strong empirical performance.
  2. The proposed Attention Perturbation is well motivated. The Attention Perturbation utilizes teacher prediction contrasts to provide HOI enhancement and product-preservation response. This component is intuitive and well aligned with the challenges of controllable HOI video generation. The paper proves its effectiveness under few-step distillation setting.
  3. The paper is clearly presented. The method is easy to follow, and the overall framework is explained in a structured manner.
Critical weaknesses
  1. The generalization ability to unseen HOI patterns and unseen object categories is not sufficiently evaluated. It remains unclear how well the method performs when the interaction type or object category is absent from the training set.

  2. The HOI evaluation relies heavily on a small test set with only 50 questions and a VLM-based metric whose reliability is not fully validated. While Qwen3.8 provides a convenient proxy for interaction plausibility, its robustness for fine-grained HOI understanding is unclear. It would strengthen the paper to include a user study or correlation analysis showing how well Qwen3.8-based judgments align with human evaluation.

  3. The paper does not discuss failure cases. Since HOI video synthesis is particularly challenging, it would be helpful to understand the common failure modes of the proposed method. A dedicated failure-case analysis would also clarify the current limitations of the approach.

  4. The data curation pipeline depends on multiple pretrained models, but the quality-control is under-specified. Because the training data is derived through several off-the-shelf tools, errors in segmentation, mesh recovery, or reference extraction could propagate into training. The paper should explain more clearly how noisy samples are filtered and how the overall data quality is ensured.

  5. The paper lacks an explicit evaluation of text-video alignment. Since prompt adherence is one of the important aspects of generated video quality, it would be useful to report a dedicated text-video alignment metric. Adding results on VideoReward [1] would make the evaluation more complete.

  6. The advantage of the proposed InteractVerse dataset is not fully reflected. The advantage of InteractVerse compared to SparseHOI-5K dataset seems to be mainly more videos, additional caption annotations and different reference images. As the caption of InteractVerse is generated via VLM and references of InteractVerse are extracted with Qwen-Image-Edit, it would be great if authors can show a baseline of distilling a few-step HOI video generator using SparseHOI-5K with generated captions and reference images.

[1] Liu., et.al. Improving Video Generation with Human Feedback. NeurIPS 2025.

Questions for authors
  1. What is the source of dataset? The paper just mentions it is from real-world e-commerce scenarios, while it is unclear whether the videos are from the public dataset or from the private sources.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zBKs

W1. Generalization to unseen interactions and object categories

FastInteract demonstrates strong generalization to unseen HOI patterns and object categories. We trained on our self-curated InteractVerse dataset and evaluated on 135 test cases from HOIVG-Bench, released by OmniShow[1]. The two datasets are independently constructed: InteractVerse contains real e-commerce livestream footage, whereas HOIVG-Bench uses AI-generated person and product references. Our quantitative and qualitative results support FastInteract’s ability to generalize across these distinct data distributions.

InteractVerse covers a wide range of products and diverse interaction types across 11 top-level product categories. Detailed category information is provided in InteractVerse Dataset Page.

W2. Evaluation scale and human validation of VLM-QA

We agree that broader evaluation coverage and validation against human judgments would strengthen the evidence. We conducted a perceptual user study with 12 evaluators. Each evaluator was shown 10 test cases randomly sampled from HOIVG-Bench. For each case, videos generated by the compared methods were presented in a blind, randomized order. Evaluators ranked the videos separately according to three criteria: Interaction Plausibility, Object Consistency, and Human/Background Consistency. We report the mean rank for each criterion, with lower values indicating better perceived quality.

User study results. Best and second-best values across all evaluated methods are bold and underlined, respectively.

MethodInteraction Plausibility ↓Object Consistency ↓Human/Background Consistency ↓
Phantom5.27505.20835.7917
VACE4.95004.41673.8167
HuMo4.71676.44176.6333
CoInteract (Teacher)4.11676.45833.6500
LiveAvatar5.69174.25005.0583
FlashTalk5.22504.35004.7583
SkyReels-V35.52505.82506.8167
LongCat-Video-Avatar 1.55.01674.2167‾\underline{4.2167}4.7667
Ours4.4833‾\underline{4.4833}3.83333.7083‾\underline{3.7083}

Even compared with multi-step models, ours outputs achieve the lowest mean rank in Object Consistency and the second-lowest mean rank in both Interaction Plausibility and Human/Background Consistency. To better understand the differences between the user-study rankings and the evaluation, we collected follow-up feedback from the evaluators.

For Interaction Plausibility, evaluators perceived the motion in our outputs more closely matched how people naturally present and interact with products, whereas some competing outputs exhibited exaggerated movements that appeared less realistic. For Object Consistency, evaluators noted oversaturated product appearances in competing outputs and preferred the overall consistency of our product appearance. For Human/Background Consistency, many evaluators noted that our outputs preserved person identity while maintaining good background consistency, faithfully reproducing the background in the reference image.

A randomly selected set of 20 qualitative cases will also be provided for inspection at User Study Page. These examples are intended to help reviewers directly assess the quality of the generated videos.

W3. Failure cases and limitations

We discuss the limitations of our method in Appendix A. We provide concrete failure cases in which the model fails to generate interactions between handheld objects and body parts other than the hands. Representative examples are available at Failure Case Page. We will incorporate these examples and a dedicated failure-case analysis into the revised appendix.

W4. Quality control in the data curation pipeline

We apply additional VLM-based verification after image editing and mesh reconstruction. For reference extraction, the VLM checks whether the extracted person and product references preserve the identity in the source video. Samples with mismatched references, missing entities, or severe visual artifacts are discarded. We also use Qwen-Omni to filter out videos with low-quality, unclear, or noisy audio. These checks reduce the propagation of preprocessing errors into training.

References

[1] Zhou et al. OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer zBKs

W5. Dedicated text–video alignment evaluation

We agree that prompt adherence warrants a dedicated evaluation in addition to the existing interaction and visual-quality metrics. Following the reviewer's suggestion, we evaluate text–video alignment using the Text Alignment (TA) score from VideoReward [1] on the existing generated videos for all 135 HOIVG-Bench cases, using consistent scoring settings and the corresponding benchmark prompts across methods.

Text–video alignment results (VideoReward TA; higher is better). Best and second-best values within each NFE group are bold and underlined, respectively.

MethodTA ↑
Many NFE
Phantom0.990266
VACE1.022103‾\underline{1.022103}
HuMo0.963820
CoInteract (Teacher)1.028622
Few NFE
LiveAvatar0.919590
FlashTalk0.892729
SkyReels-V30.816455
LongCat-Video-Avatar 1.50.995104
Ours (Student)0.987383‾\underline{0.987383}

We observe that few-step models generally exhibit weaker prompt adherence than many-step models. Within the few-step group, our four-step model achieves a TA score comparable to that of the best-performing model.

W6. Value of InteractVerse relative to an augmented SparseHOI-5K

The main limitation of SparseHOI-5K is its restricted identity and product coverage: its 4,850 clips contain only 34 person identities and approximately 1,000 product reference images. Generating additional captions and extracting reference images do not expand the underlying set of people, products, or recorded interactions. In contrast, InteractVerse contains 37,477 clips spanning 53 product categories and diverse interaction types, with details available on our InteractVerse Dataset Page. The clips are collected from different e-commerce livestreams at different times, providing broad coverage of person identities and products. After excluding a small number of repeated products and person identities, the dataset contains over 30,000 distinct pairs of product and person reference images. Our LiveAvatar competitor uses approximately 400,000 training samples for distillation, whereas SparseHOI-5K contains fewer than 5,000 clips. Based on this distillation experience, we consider SparseHOI-5K insufficient in both size and diversity to meet the needs of our distillation task.

Q1. Source of the dataset

The source videos were collected by us from publicly accessible e-commerce livestreaming platforms. InteractVerse was newly curated from these source videos. The collected footage is processed into interaction segments and annotated through the pipeline described in Section 3 and Appendix B. We will explicitly state this provenance in the revised manuscript.

References

[1] Liu et al. Improving Video Generation with Human Feedback. NeurIPS 2025.

I would like to thank authors' efforts for trying to address my concerns. My concerns about failure case analysis and text-video alignment are addressed, but I still encourage authors to add more failure case and limitation discussions so that people can understand where the challenge is. Below are my remaining concerns:

W1: My concerns about the generalization still remain unaddressed. The linked qualitative results mainly show bottle-shaped objects with similar interaction patterns, providing limited evidence of generalization to diverse objects and interactions. Given the broad coverage of e-commerce categories in training, it is unclear whether the benchmarks evaluate genuinely unseen categories or interaction patterns. Please clarify this coverage and provide more diverse examples, ideally including explicitly held-out scenarios.

W2: The VLM-QA based HOI quality measurement might be vulnerable. LiveAvatar ranks second on VLM-QA in the Few-NFE setting but last in the human evaluation. It also obtains similar VLM-QA scores to LongCat-Video-Avatar 1.5 despite substantially different human rankings. I tend to consider VLM-QA evaluation on only 50 questions in the paper cannot reflect the HOI quality of each HOI video generation method.

W4: Unclear robustness of VLM-based Quality control The data curation pipeline fully rely on VLM-based Quality control. It is unclear whether the training samples contain any artifacts and whether they are aligned with human judgments. A human audit of a representative subset would provide stronger evidence of the curated dataset’s quality.

W6: The value of InteractVerse dataset is still not very clear to me. I also expect augmented SparseHOI-5K cannot produce a good distilled student, but as SparseHOI-5K also contains videos from e-commerce platforms, it may already cover many of the tested scenarios and might be sufficient to deal with your benchmarks. A controlled distillation comparison using the training protocol, together with a clear analysis of benchmark coverage, would help strengthen the dataset’s added value.

AuthorsOpenReview ↗

Response to Reviewer zBKs

W1: Concerns about the generalization remain unaddressed. Our model also supports interactions with a much broader range of product. We additionally provide qualitative comparisons involving several non-bottle-shaped objects on the More Cases Page. For interaction types, it's controlled by the text prompt and product: changing them yield different interaction types, and the interaction types in the benchmark differ from those in our InteractVerse training set.

As for object categories, our evaluation setting follows the same protocol as that used to evaluate the teacher model: relative to the training set, the test set does not contain unseen categories. In addition, our paper focuses on the distilled model, whose generalization is largely inherited from the teacher model; requiring the distilled model to achieve stronger category generalization is therefore not the primary goal of this paper, and we will investigate this direction further in future work.

W2: The VLM-QA based HOI quality measurement might be vulnerable.

We have examined the LiveAvatar outputs directly. In many of its videos, the hand that is not engaged in the interaction is malformed, showing fused fingers or implausible joint structure, while the interacting hand and the contact region remain acceptable. Human evaluators notice this immediately, since a malformed hand is visually salient regardless of which hand it is. VLM-QA questions, by contrast, are scoped to the hand–object contact region, so a defect on the non-interacting hand contributes little to the score, and this is the case where the gap appears. Representative cases are available at User Study Page, where the malformed non-interacting hand is visible in the LiveAvatar column.

We compare the method ordering induced by VLM-QA with that induced by user study ranks. The following table reports the correlations after aligning score directions. Here, n denotes the number of methods.

SettingnSpearman's rhoKendall's tauPearson's r
All methods90.733 (p = 0.025)0.556 (p = 0.045)0.770 (p = 0.015)
Excluding LiveAvatar80.857 (p = 0.007)0.714 (p = 0.014)0.829 (p = 0.011)

These results provide evidence of agreement at the aggregate method level. The individual method ranks are shown below. Rank 1 is best. The rank difference is calculated by subtracting the human rank from the VLM-QA rank.

MethodVLM-QA rankHuman rankRank difference
Phantom770
VACE34-1
HuMo13-2
CoInteract (Teacher)21+1
LiveAvatar59-4
FlashTalk86+2
SkyReels-V398+1
LongCat-Video-Avatar 1.565+1
Ours42+2

For eight of the nine methods, the VLM-QA and human rankings differ by no more than two positions. LiveAvatar is the only exception, with a four-position difference.

Excluding LiveAvatar, the above results indicate that VLM-QA can reflect the trend of human perception (Spearman's ρ = 0.857, p = 0.007; Kendall's τ = 0.714, p = 0.014).

W4: Unclear robustness of VLM-based Quality control. Our data filtering process does not rely entirely on VLM filtering. Because each filtering stage poses a different verification task, and VLM filtering requires task-specific prompts and hyperparameters for each, we validated every VLM deployment separately before applying it at scale. For each deployment, we first randomly sampled 100 clips and tuned the prompt and hyperparameters on this subset. We then sampled another independent set of 100 clips to evaluate the tuned configuration, checking whether the VLM correctly retained the intended samples and rejected those containing artifacts, missing entities, mismatched references, or ambiguous correspondence. Only after this verification did we deploy the filter on the full-scale dataset. The curation process therefore combines automated filtering with human review at every stage, rather than relying on the VLM alone. We will include this discussion in the revised manuscript.

W6: The value of the InteractVerse dataset. The SparseHOI-5K dataset has not yet been released, so we can only estimate its product coverage and data quality from its paper. For distillation tasks, dataset value is not determined solely by the number of clips or hours, but also by the diversity and coverage of the underlying data distribution. In particular, DMD-based distillation requires sufficient coverage to support effective distribution matching. The 4,850 clips (approximately 10 hours) reported for SparseHOI-5K are unlikely to provide sufficient coverage required by DMD. Once it is released, we will conduct a detailed comparison with our dataset.

Comments on the paper

1 comment

Thank you all for taking the time to read our paper carefully and share your feedback. In our responses, we have discussed your comments in detail and included results from additional experiments. We have also provided anonymous links to selected examples and would be grateful if you could take a look at them. We hope these results and clarifications address your concerns and make the paper’s contributions clearer. We would appreciate it if you could consider them when revisiting your assessment. We would be happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.