- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper presents PhysHuman, a physics-aware framework that jointly generates human-object interaction (HOI) videos and actuation maps representing joint torque magnitudes and directions from a reference image and a text prompt. The paper has two main contributions. First, it introduces PhysHuman-Dataset, containing over 15,000 simulated HOI video clips with varying object mass and friction. Second, it proposes Actuation Maps based on human motion dynamics and uses them as auxiliary generation targets to improve the physical plausibility of generated videos. Experiments compare PhysHuman with existing video generation models using visual quality metrics, physics-related metrics, and a user study. Ablation studies evaluate different model designs and auxiliary representations, including optical flow and depth. The reported results show improvements in video quality and physical plausibility.
Critical strengths
- The paper introduces body-aligned actuation maps that represent human joint torque magnitudes and directions. Using these maps as auxiliary supervision for video generation is an interesting idea and shows potential for improving the physical plausibility of generated HOI videos.
- The paper constructs a synthetic HOI dataset containing over 15,000 video clips with simulated joint actuation annotations under varying object mass and friction conditions. This dataset could provide a useful resource for studying physics-aware HOI generation.
Critical weaknesses
- Beyond introducing Actuation Maps as auxiliary generation targets, the proposed framework largely relies on existing techniques, including Wan2.2 and temporal concatenation. The method simply incorporates Actuation Maps into the joint diffusion process without additional physics-specific loss functions, explicit physical constraints, or dedicated mechanisms to ensure consistency between generated motions and joint torques. Therefore, the methodological novelty is limited, and it remains unclear to what extent the model learns physically meaningful dynamics.
- Although the paper claims joint video and dynamics generation, it does not quantitatively evaluate the predicted Actuation Maps or their physical consistency with the generated motions. Furthermore, the main experiments compare PhysHuman, which is fine-tuned on the proposed dataset, against zero-shot video generation baselines without target-domain fine-tuning. The paper and supplementary material also lack qualitative comparisons with these baselines. These issues substantially weaken the experimental evidence supporting the paper's main claims.
Questions for authors
- The paper introduces PHD as a physics-grounded dataset with varying object mass and friction. However, the ranges, sampling strategies, and distributions of these physical parameters are not clearly specified. Could the authors provide these details, including how many physical configurations are generated per reference motion? Additionally, how is the accuracy of the simulation-based motion tracking evaluated? These details are important for assessing the physical reliability, diversity, and reproducibility of the dataset.
- The paper uses PSNR, SSIM, and LPIPS to compare generated videos against the ground-truth sequence. However, HOI video generation conditioned on a first frame and a text prompt is inherently a one-to-many problem, where multiple valid motion trajectories and timings may satisfy the same conditions. How do the authors justify using these reference-based metrics, which may penalize physically plausible generations that differ from the ground truth? Could the authors clarify how temporal alignment and generation randomness are handled during evaluation?
- The current experiments do not clearly show whether the improvements actually come from the physical information in Actuation Maps. For example, the proposed method generates both Magnitude and Direction Maps, while the depth and optical flow baselines use only one auxiliary representation. Are these comparisons conducted under comparable training settings? Could the authors also include controls using body-part maps or shuffled actuation maps to verify whether the improvements are truly due to joint torque information?
7 comments
Response to Reviewer unEz(2/5)
W2(a). Quantitative evaluation of the actuation maps
We thank the reviewer for highlighting this missing evaluation. We agree that visually coherent actuation maps do not necessarily imply physically meaningful predictions, and that quantitative validation is essential.
We therefore introduce a simulation-based actuation–motion consistency evaluation. A direct comparison against the original ground-truth actuation maps would be affected by the one-to-many nature of video generation: the generated interaction may follow a different valid motion trajectory or timing. Instead, we recover 3D human–object motion from each generated video and track it using the same Isaac Gym/InterMimic simulation framework employed in constructing PHD. We then compare the generated actuation-magnitude maps with the joint-effort signals obtained from physics-based tracking.
Evaluation protocol:
- Motion recovery. We recover the 3D human motion from the generated video with GVHMR (Shen et al., SIGGRAPH Asia 2024) and place it in the simulation world with the known camera of the conditioning frame. Each frame is shifted vertically so that the lowest foot touches the ground. The 6-DoF object trajectory is recovered from the same video by segmenting the object with SAM 2 (Ravi et al., ICLR 2025) and fitting the object mesh to its silhouettes. The 15 fps motion is interpolated to the 30 Hz control rate.
- Physics-based tracking. The recovered human and object motion is the reference for the same simulator (Isaac Gym) and tracking controller (InterMimic) that we used to build PHD, with the object density of the clip. The controller drives the humanoid through PD-controlled joints (physics at 60 Hz, control at 30 Hz). No external force assists the body, so the joint torques it applies are the actuation that the generated motion physically requires.
- Tracking failures. If the mean key-body error exceeds 25 cm, for example on a physically implausible generated motion, the humanoid and the object are reset to the reference state of that frame and tracking continues. Frames from 2 before to 10 after a reset are excluded from the scores, because a reset causes a torque transient.
- Frame alignment. Generated frame k and simulation frame k are the same instant, so the comparison is frame by frame, without temporal warping.
Comparison. We compare magnitude-map values extracted over nine body regions with the corresponding tracked joint-effort magnitudes. Across-part Spearman correlation measures agreement in relative effort within each frame; per-part temporal Pearson correlation measures agreement in variation over time. We apply a fixed 67 ms offset selected using ground-truth videos, without per-clip temporal alignment, and use circularly shifted sequences as a temporal control. Frames around tracking resets are excluded, and clips with fewer than 20 valid frames are omitted. This retains 50 GT clips and 46 generated clips. We will include the full preprocessing, tracking-error, exclusion, and alignment specifications in the revision.
Table R2. Actuation-magnitude consistency with motion-derived tracking references. Values are mean ± standard error over clips.
| Per-frame relative effort ↑ | Temporal alignment ↑ | Temporal chance level | p (vs. chance) | |
|---|---|---|---|---|
| Ground-truth video + simulated map (upper bound) | 0.663 ± 0.025 | 0.339 ± 0.032 | −0.100 | 8 × 10⁻¹² |
| PhysHuman: generated video + generated map | 0.618 ± 0.033 | 0.118 ± 0.028 | −0.040 | 7 × 10⁻⁶ |
As is demonstrated in the table above, the generated magnitude maps achieve an across-part correlation of 0.618, compared with 0.663 for the GT-video reference. Temporal correlation is modest but exceeds the circular-shift control (paired Wilcoxon signed-rank test over clips, ).
These results indicate that the generated magnitude maps capture meaningful differences in relative effort across body regions, although temporal correspondence remains more limited. This evaluation does not require alignment between generated and original ground-truth trajectories. It measures magnitude consistency under the tracking simulator, rather than exact torque recovery or directional accuracy.
Response to Reviewer unEz(3/5)
W2(b). Fairness of baseline comparisons
We appreciate the reviewer's suggestion to include target-domain fine-tuned baselines. We agree that this is important for distinguishing improvements from domain adaptation versus those associated with actuation supervision.
The original submission already includes an RGB-only Wan2.2 baseline fine-tuned on the same PHD dataset (Table 3). This provides a controlled within-backbone comparison, where PhysHuman improves FVD from 287.37 to 184.31.
To further strengthen the comparison, we additionally fine-tune CogVideoX1.5-5B-I2V and HunyuanVideo-I2V on PHD.
Training protocol. The additional baselines use the same target-domain training clips, conditioning inputs (first frame and text prompt), and evaluation resolution (480 × 832), with 81-frame videos at 15 fps. Each follows its corresponding LoRA fine-tuning recipe, with a training-compute budget capped at that of PhysHuman. We evaluate checkpoints every 250 steps on a separate 20-clip validation set with held-out reference motions.
Table R4. Comparison with PHD-fine-tuned video-generation baselines (50 test clips).
| Method | Fine-tuned on PHD | FID ↓ | FVD ↓ | Foot sliding ↓ | Foot Ground ↓ | Jitter (m/s³) ↓ | Ground penetration ↓ |
|---|---|---|---|---|---|---|---|
| CogVideoX1.5-5B-I2V | ✓ | 79.41 | 429.53 | 21.42 | 9.13 | 80.9 | 11.08 |
| HunyuanVideo-I2V | ✓ | 66.38 | 251.69 | 10.12 | 8.79 | 30.5 | 5.54 |
| Wan2.2 (RGB-only) | ✓ | 70.14 | 287.37 | 12.70 | 11.52 | 62.0 | 5.51 |
| PhysHuman | ✓ | 53.76 | 184.31 | 9.76 | 6.41 | 37.7 | 3.96 |
PhysHuman achieves the best FVD, foot-sliding, foot-ground and ground-penetration scores among the fine-tuned models. HunyuanVideo achieves a lower joint-jitter score, potentially benefiting from stronger physical priors learned by its larger, 13B-parameter backbone.
Together with the within-backbone RGB-only comparison, these results provide stronger evidence that the improvements are not solely attributable to fine-tuning on the target domain.
Response to Reviewer unEz(1/5)
We thank the reviewer for the thoughtful assessment and specific suggestions. We appreciate the recognition of our actuation-map representation and the value of PHD. The reviewer raises several important questions regarding methodological novelty, the physical meaning of generated actuation, and the fairness of the evaluation. These comments motivated us to conduct additional experiments that more directly examine the contribution of torque-derived supervision.
W1. Methodological novelty and physically meaningful dynamics
We appreciate the reviewer's observation that PhysHuman builds on established video-generation architectures. This is an important distinction, and we would like to clarify that our primary contribution is introducing physically grounded actuation as a learnable auxiliary generation target, rather than proposing a new diffusion backbone.
Specifically, our contributions are:
We agree that an important question is whether the observed improvements genuinely arise from physical information, rather than simply adding auxiliary tokens or body-structure supervision. To investigate this, we provide two additional pieces of evidence:
Importantly, our approach learns physics-related motion priors through auxiliary supervision, rather than imposing hard physical constraints. We appreciate the suggestion regarding explicit physical consistency mechanisms, which we see as a promising complementary direction. Our new experiments aim to clarify the extent to which physics-derived supervision alone contributes to generation quality.