acceptodds
Under review as a conference paper at ICLR 2027

PhysHuman: Physics Aware Human and Object Interaction Video and Dynamics Joint Generation

Abstract

Human–object interaction (HOI) video generation promises scalable creation of interactive human motions for graphics and embodied applications. However, despite rapid progress, most existing models primarily optimize for RGB fidelity while overlooking physical plausibility and actionable physical signals that govern human motion, often producing interpenetrations, implausible impulses, and motions that fail to vary coherently with physical settings and interactions. To tackle this issue, we propose PhysHuman, a physics-informed approach that jointly generates HOI video and dynamics by conditioning a diffusion-based video model on text prompt and first frame. By predicting actuator torque as an auxiliary target, the model learns structured dynamic information, thereby improving the overall quality of the generated video without providing actuator torque information . Furthermore, to facilitate learning and evaluation of physics-informed HOI generation, we introduce PhysHuman-Dataset (PHD), a controllable corpus constructed in simulation by executing HOI policies while sweeping object mass, providing synchronized RGB, metrically accurate depth and actuation traces, together with geometry and physical-parameter annotations. Across various evaluations, PhysHuman improves physics consistency while simultaneously unifying the generation and perception of HOI.

Primary area:
generative models
Keywords:
physics simulation,video generation,actuation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_unEzOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper presents PhysHuman, a physics-aware framework that jointly generates human-object interaction (HOI) videos and actuation maps representing joint torque magnitudes and directions from a reference image and a text prompt. The paper has two main contributions. First, it introduces PhysHuman-Dataset, containing over 15,000 simulated HOI video clips with varying object mass and friction. Second, it proposes Actuation Maps based on human motion dynamics and uses them as auxiliary generation targets to improve the physical plausibility of generated videos. Experiments compare PhysHuman with existing video generation models using visual quality metrics, physics-related metrics, and a user study. Ablation studies evaluate different model designs and auxiliary representations, including optical flow and depth. The reported results show improvements in video quality and physical plausibility.

Critical strengths
  1. The paper introduces body-aligned actuation maps that represent human joint torque magnitudes and directions. Using these maps as auxiliary supervision for video generation is an interesting idea and shows potential for improving the physical plausibility of generated HOI videos.
  2. The paper constructs a synthetic HOI dataset containing over 15,000 video clips with simulated joint actuation annotations under varying object mass and friction conditions. This dataset could provide a useful resource for studying physics-aware HOI generation.
Critical weaknesses
  1. Beyond introducing Actuation Maps as auxiliary generation targets, the proposed framework largely relies on existing techniques, including Wan2.2 and temporal concatenation. The method simply incorporates Actuation Maps into the joint diffusion process without additional physics-specific loss functions, explicit physical constraints, or dedicated mechanisms to ensure consistency between generated motions and joint torques. Therefore, the methodological novelty is limited, and it remains unclear to what extent the model learns physically meaningful dynamics.
  2. Although the paper claims joint video and dynamics generation, it does not quantitatively evaluate the predicted Actuation Maps or their physical consistency with the generated motions. Furthermore, the main experiments compare PhysHuman, which is fine-tuned on the proposed dataset, against zero-shot video generation baselines without target-domain fine-tuning. The paper and supplementary material also lack qualitative comparisons with these baselines. These issues substantially weaken the experimental evidence supporting the paper's main claims.
Questions for authors
  1. The paper introduces PHD as a physics-grounded dataset with varying object mass and friction. However, the ranges, sampling strategies, and distributions of these physical parameters are not clearly specified. Could the authors provide these details, including how many physical configurations are generated per reference motion? Additionally, how is the accuracy of the simulation-based motion tracking evaluated? These details are important for assessing the physical reliability, diversity, and reproducibility of the dataset.
  2. The paper uses PSNR, SSIM, and LPIPS to compare generated videos against the ground-truth sequence. However, HOI video generation conditioned on a first frame and a text prompt is inherently a one-to-many problem, where multiple valid motion trajectories and timings may satisfy the same conditions. How do the authors justify using these reference-based metrics, which may penalize physically plausible generations that differ from the ground truth? Could the authors clarify how temporal alignment and generation randomness are handled during evaluation?
  3. The current experiments do not clearly show whether the improvements actually come from the physical information in Actuation Maps. For example, the proposed method generates both Magnitude and Direction Maps, while the depth and optical flow baselines use only one auxiliary representation. Are these comparisons conducted under comparable training settings? Could the authors also include controls using body-part maps or shuffled actuation maps to verify whether the improvements are truly due to joint torque information?
7 comments
AuthorsOpenReview ↗

Response to Reviewer unEz(1/5)

We thank the reviewer for the thoughtful assessment and specific suggestions. We appreciate the recognition of our actuation-map representation and the value of PHD. The reviewer raises several important questions regarding methodological novelty, the physical meaning of generated actuation, and the fairness of the evaluation. These comments motivated us to conduct additional experiments that more directly examine the contribution of torque-derived supervision.

W1. Methodological novelty and physically meaningful dynamics

We appreciate the reviewer's observation that PhysHuman builds on established video-generation architectures. This is an important distinction, and we would like to clarify that our primary contribution is introducing physically grounded actuation as a learnable auxiliary generation target, rather than proposing a new diffusion backbone.

Specifically, our contributions are:

  1. Actuation maps: A body-aligned representation that converts simulated joint torques into image-space supervision, allowing pretrained video models to jointly learn appearance and dynamics without introducing modality-specific architectures.
  2. PhysHuman-Dataset (PHD): A physics-grounded HOI dataset providing synchronized RGB videos, simulated actuation, and physical parameters, enabling systematic studies of physics-aware video generation.

We agree that an important question is whether the observed improvements genuinely arise from physical information, rather than simply adding auxiliary tokens or body-structure supervision. To investigate this, we provide two additional pieces of evidence:

  • Torque-information controls (see below in Table R7): We introduce body-part-map and shuffled-actuation controls with the same auxiliary-stream count as PhysHuman. The correctly associated actuation representation achieves substantially better FVD (184.31) than either body-part maps (277.92) or shuffled actuation (252.48), along with lower joint jitter. This suggests that the improvements cannot be fully explained by body-layout information or additional auxiliary tokens alone.
  • Actuation–motion consistency (Table R2): We further evaluate whether generated actuation corresponds to the motion in the generated RGB video. The generated magnitude maps achieve an across-body-part Spearman correlation of 0.618 with simulator-derived actuation references, compared with 0.663 for ground-truth videos. This provides quantitative evidence of meaningful relative-effort correspondence.

Importantly, our approach learns physics-related motion priors through auxiliary supervision, rather than imposing hard physical constraints. We appreciate the suggestion regarding explicit physical consistency mechanisms, which we see as a promising complementary direction. Our new experiments aim to clarify the extent to which physics-derived supervision alone contributes to generation quality.

AuthorsOpenReview ↗

Response to Reviewer unEz(2/5)

W2(a). Quantitative evaluation of the actuation maps

We thank the reviewer for highlighting this missing evaluation. We agree that visually coherent actuation maps do not necessarily imply physically meaningful predictions, and that quantitative validation is essential.

We therefore introduce a simulation-based actuation–motion consistency evaluation. A direct comparison against the original ground-truth actuation maps would be affected by the one-to-many nature of video generation: the generated interaction may follow a different valid motion trajectory or timing. Instead, we recover 3D human–object motion from each generated video and track it using the same Isaac Gym/InterMimic simulation framework employed in constructing PHD. We then compare the generated actuation-magnitude maps with the joint-effort signals obtained from physics-based tracking.

Evaluation protocol:

  1. Motion recovery. We recover the 3D human motion from the generated video with GVHMR (Shen et al., SIGGRAPH Asia 2024) and place it in the simulation world with the known camera of the conditioning frame. Each frame is shifted vertically so that the lowest foot touches the ground. The 6-DoF object trajectory is recovered from the same video by segmenting the object with SAM 2 (Ravi et al., ICLR 2025) and fitting the object mesh to its silhouettes. The 15 fps motion is interpolated to the 30 Hz control rate.
  2. Physics-based tracking. The recovered human and object motion is the reference for the same simulator (Isaac Gym) and tracking controller (InterMimic) that we used to build PHD, with the object density of the clip. The controller drives the humanoid through PD-controlled joints (physics at 60 Hz, control at 30 Hz). No external force assists the body, so the joint torques it applies are the actuation that the generated motion physically requires.
  3. Tracking failures. If the mean key-body error exceeds 25 cm, for example on a physically implausible generated motion, the humanoid and the object are reset to the reference state of that frame and tracking continues. Frames from 2 before to 10 after a reset are excluded from the scores, because a reset causes a torque transient.
  4. Frame alignment. Generated frame k and simulation frame k are the same instant, so the comparison is frame by frame, without temporal warping.

Comparison. We compare magnitude-map values extracted over nine body regions with the corresponding tracked joint-effort magnitudes. Across-part Spearman correlation measures agreement in relative effort within each frame; per-part temporal Pearson correlation measures agreement in variation over time. We apply a fixed 67 ms offset selected using ground-truth videos, without per-clip temporal alignment, and use circularly shifted sequences as a temporal control. Frames around tracking resets are excluded, and clips with fewer than 20 valid frames are omitted. This retains 50 GT clips and 46 generated clips. We will include the full preprocessing, tracking-error, exclusion, and alignment specifications in the revision.

Table R2. Actuation-magnitude consistency with motion-derived tracking references. Values are mean ± standard error over clips.

Per-frame relative effort ↑Temporal alignment ↑Temporal chance levelp (vs. chance)
Ground-truth video + simulated map (upper bound)0.663 ± 0.0250.339 ± 0.032−0.1008 × 10⁻¹²
PhysHuman: generated video + generated map0.618 ± 0.0330.118 ± 0.028−0.0407 × 10⁻⁶

As is demonstrated in the table above, the generated magnitude maps achieve an across-part correlation of 0.618, compared with 0.663 for the GT-video reference. Temporal correlation is modest but exceeds the circular-shift control (paired Wilcoxon signed-rank test over clips, p=7×10−6p =7\times 10^{-6} ).

These results indicate that the generated magnitude maps capture meaningful differences in relative effort across body regions, although temporal correspondence remains more limited. This evaluation does not require alignment between generated and original ground-truth trajectories. It measures magnitude consistency under the tracking simulator, rather than exact torque recovery or directional accuracy.

AuthorsOpenReview ↗

Response to Reviewer unEz(3/5)

W2(b). Fairness of baseline comparisons

We appreciate the reviewer's suggestion to include target-domain fine-tuned baselines. We agree that this is important for distinguishing improvements from domain adaptation versus those associated with actuation supervision.

The original submission already includes an RGB-only Wan2.2 baseline fine-tuned on the same PHD dataset (Table 3). This provides a controlled within-backbone comparison, where PhysHuman improves FVD from 287.37 to 184.31.

To further strengthen the comparison, we additionally fine-tune CogVideoX1.5-5B-I2V and HunyuanVideo-I2V on PHD.

Training protocol. The additional baselines use the same target-domain training clips, conditioning inputs (first frame and text prompt), and evaluation resolution (480 × 832), with 81-frame videos at 15 fps. Each follows its corresponding LoRA fine-tuning recipe, with a training-compute budget capped at that of PhysHuman. We evaluate checkpoints every 250 steps on a separate 20-clip validation set with held-out reference motions.

Table R4. Comparison with PHD-fine-tuned video-generation baselines (50 test clips).

MethodFine-tuned on PHDFID ↓FVD ↓Foot sliding ↓Foot Ground ↓Jitter (m/s³) ↓Ground penetration ↓
CogVideoX1.5-5B-I2V✓79.41429.5321.429.1380.911.08
HunyuanVideo-I2V✓66.38251.6910.128.7930.55.54
Wan2.2 (RGB-only)✓70.14287.3712.7011.5262.05.51
PhysHuman✓53.76184.319.766.4137.73.96

PhysHuman achieves the best FVD, foot-sliding, foot-ground and ground-penetration scores among the fine-tuned models. HunyuanVideo achieves a lower joint-jitter score, potentially benefiting from stronger physical priors learned by its larger, 13B-parameter backbone.

Together with the within-backbone RGB-only comparison, these results provide stronger evidence that the improvements are not solely attributable to fine-tuning on the target domain.

Reviewer_eNSXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce PhysHuman, a DIT model that given an initial image and text prompt, generates human-object interactions (HOI) videos along with body-aligned maps that represent joint actuation magnitued and direction. On the other hand, the authors also introduce PHD a dataset with over 15K simulated and rendered interaction clips with motion, actuation and physical-parameter annotations. Results show improved video genration, while the accuracy of the generation actuation maps remains unverified.

Critical strengths

Strengths:

  • The proposed dataset (PHD) provides useful supervision that ordinary videos lack. PHD has pairs of human-object interactions with joint efforts and physical parameters, which supports studying how physical supervision can improve video generation and is a useful contribution beyond the model itself.

  • The authors provide controlled comparisons which support the benefit of the actuation supervision. Compared with an RGB-only model that was fine-tuned on the same dataset PhysHuman improved FVD, it also achieves better foot-sliding, hitter and ground-penetration scores than the flow and depth variants.

Critical weaknesses

Weaknesses

  • I see that the generated actuation maps are not quantitatively validated, the paper shows that the maps look coherent, but does not measure their accuracy against the available synthetic gt or check whether they agree with the generated motion.

  • I think that the evaluation leaves uncertainty about generalization and physical improvement. The test set contains only 50 clips, and Appendix B does not establish whether variants of the same reference motion appear in both training and testing, this matters because the dataset creates multiple physical and visual variants from reference sequences, on the other hand, the RGB-only fine-tuned baseline is absent from the physical evaluation and user study.

Questions for authors

Please refer to weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer eNSX(1/2)

We thank the reviewer for recognising the value of PHD and of the controlled RGB-only comparison.

W1. The generated actuation maps are not quantitatively validated

We agree that visual coherence alone is insufficient to validate the generated actuation. We therefore add a simulation-based consistency evaluation of the generated magnitude maps. Direct comparison with the original ground-truth maps would confound actuation errors with differences in generated poses and timing. Instead, we compare the effort indicated by each generated map with actuation signals obtained by tracking motion recovered from its corresponding RGB video.

Evaluation protocol:

  1. Motion recovery. We recover the 3D human motion from the generated video with GVHMR (Shen et al., SIGGRAPH Asia 2024) and place it in the simulation world with the known camera of the conditioning frame. Each frame is shifted vertically so that the lowest foot touches the ground. The 6-DoF object trajectory is recovered from the same video by segmenting the object with SAM 2 (Ravi et al., ICLR 2025) and fitting the object mesh to its silhouettes. The 15 fps motion is interpolated to the 30 Hz control rate.
  2. Physics-based tracking. The recovered human and object motion is the reference for the same simulator (Isaac Gym) and tracking controller (InterMimic) that we used to build PHD, with the object density of the clip. The controller drives the humanoid through PD-controlled joints. No external force assists the body, so the joint torques it applies are the actuation that the generated motion physically requires.
  3. Tracking failures. If the mean key-body error exceeds 25 cm, for example on a physically implausible generated motion, the humanoid and the object are reset to the reference state of that frame and tracking continues.
  4. Frame alignment. Generated frame k and simulation frame k are the same instant, so the comparison is frame by frame, without temporal warping.

Comparison: We compare magnitude-map values extracted over nine body regions with the corresponding tracked joint-effort magnitudes.

  1. per-frame-relative effort:Across-part Spearman correlation measures agreement in relative effort within each frame.
  2. temporal alignment:per-part temporal Pearson correlation measures agreement in variation over time. We apply a fixed 67 ms offset selected using ground-truth videos(due to common pd-controlled tracker latency), without per-clip temporal alignment. We will include the full preprocessing, tracking-error, exclusion, and alignment specifications in the revision.

Table R2. Actuation-magnitude consistency with motion-derived tracking references. Values are mean ± standard error over clips.

Per-frame relative effort ↑Temporal alignment ↑Temporal chance levelp (vs. chance)
Ground-truth video + simulated map (upper bound)0.663 ± 0.0250.339 ± 0.032−0.1008 × 10⁻¹²
PhysHuman: generated video + generated map0.618 ± 0.0330.118 ± 0.028−0.0407 × 10⁻⁶

As is demonstrated in the table above, the generated magnitude maps achieve an across-part correlation of 0.618, compared with 0.663 for the GT-video reference. Temporal correlation is modest but exceeds the circular-shift control (paired Wilcoxon signed-rank test over clips, p=7×10−6p =7\times 10^{-6} ).

These results provide evidence of agreement in relative effort across body parts, with more limited agreement in temporal variation. This evaluation measures magnitude consistency on trackable intervals; Since the task is one-to-many, per-frame relative effort is more informative as it tells which body parts carry the load. It does not establish absolute torque accuracy.

AuthorsOpenReview ↗

Response to Reviewer eNSX(2/2)

W2. Generalization, overlap of reference motions, and the RGB-only baseline in the physical evaluation

Reference-motion overlap and test-set size.

PHD is split at the clip level. A clip is one simulation of a reference motion under one physical configuration (object density and friction), rendered once with its own camera, scene, clothing and lighting. The 50 test clips come from 50 independent reference motions , covering all 13 objects and 15 of the 17 subjects. 35 of motions also occur in the training set , while 15 of the motions never occur in the training set. The same holds for the controls in Table R7. Our simulations show that varying object density can substantially alter the resulting motion, as the reference guides but does not uniquely determine the simulated trajectory. Thus, exposure to the same reference motion during training does not guarantee successful generation under different physical configurations at test time.

RGB-only baseline in the physical evaluation. We add the RGB-only model fine-tuned on PHD with the same backbone to the physical evaluation of the original Table 4, using the same metric definitions and evaluation protocol. Significance is from a paired bootstrap over clips (10,000 resamples).

Table R3. Physical plausibility: PhysHuman versus RGB-only fine-tuning.

MethodFoot sliding (mm/frame) ↓Joint jitter (m/s³) ↓Foot–ground distance |h| (cm) ↓Ground penetration (cm) ↓
RGB-only12.7062.011.525.51
PhysHuman9.7637.76.413.96

PhysHuman numerically improves all four metrics. Under the paired bootstrap analysis over clips (10,000 resamples), improvements in joint jitter and foot–ground deviation are significant (p<.01p<.01); foot sliding has p=.08p=.08, and ground penetration does not show a statistically significant difference. The clearest improvements over RGB-only fine-tuning are therefore in motion smoothness and alignment with the ground.

CriterionPhysHuman (%)RGB-only (%)
Video quality85.714.3
Physical plausibility87.312.7
Text alignment77.722.3
Motion quality86.713.3
Average84.415.6
Reviewer_kohUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the issue of physical implausibility of HOI video generation. The proposed method jointly generates HOI video and dynamics by conditioning a diffusion-based video model on text prompt and first frame. It further predicts torque-derived actuation maps as an auxiliary target to learn structured dynamics. It also contributes PhysHuman-Dataset to this task evaluation and learning.

Critical strengths
  • The introduction of auxiliary actuation information to provide structure dynamics is interesting and expected to benefit the physical plausibility of the generated video, which is validated in the experiments.
  • It makes good contribution with a solid method and a dataset though the data is collected by simulation.
Critical weaknesses
  • The motivation to represent actuation information as maps with the same format as RGB is to avoid training VAE encoder. However, such representation may not be cost-effective. Other alternatives should be discussed.
  • The concatenation of features along the temporal dimension is expected to increase the input length to PhysHuman DiT, which would needs more memory. It is worth to present the resources required along with the performance.
Questions for authors

see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kohU

We thank the reviewer for the positive assessment and the suggestions regarding representation choices and computational cost.

W1. Is an RGB-format actuation map cost-effective? What are the alternatives?

Thanks the reviewer for pointing this out. Our choice of RGB actuation map balances compatibility with the pretrained video model, spatial correspondence, and computational cost.

Why body-aligned maps? The magnitude and direction maps use the existing Wan2.2 VAE and preserve the pretrained DiT’s input and output interfaces, without introducing a modality-specific encoder or decoder. Each joint’s actuation value is assigned to its associated body region using SMPL skinning weights and rendered with the same camera as the RGB video. This provides explicit image-space correspondence between appearance and actuation that the model can exploit during joint denoising. At inference, PhysHuman requires only the initial image and text prompt; no explicit pose or camera estimate is supplied.

Alternative representations. We considered several alternatives. (1) Compact per-joint torque vectors are a reasonable alternative that could reduce the auxiliary sequence length. Such a design would need an interface for embedding and predicting joint signals and associating them with image locations. (2) A dedicated actuation VAE could provide a more specialized encoding but requires an additional training stage. (3) We also evaluated channel concatenation of the same RGB, magnitude, and direction modalities in the original submission. This keeps the temporal sequence shorter but requires expanding the DiT’s input and output layers. We evaluated this last alternative under the same training setup (main paper Table 3), where temporal concatenation achieves substantially better FVD (184.31 vs. 305.33).

Our design thus prioritizes compatibility with the pretrained backbone and spatially aligned supervision, while accepting the additional computational cost quantified in Table R1.

W2. Resources required by temporal concatenation

We agree that jointly denoising three modality streams introduces additional computational cost. We therefore profile PhysHuman and the RGB-only model fine-tuned on PHD under the same hardware and execution settings, using 81 frames at 480 × 832 resolution.

Table R1. Resource comparison between RGB-only fine-tuning and PhysHuman. FVD values are from Table 3 of the original submission.

Video tokens / sampleTraining memory / GPU (GB) ↓Training time / step (s) ↓Inference time / video (s) ↓FVD ↓
RGB-only: Wan2.2 fine-tuned on PHD8,19031.46.034.3287.37
PhysHuman25,35034.616.0132.3184.31

Measured on RTX PRO 6000 with the training recipe of the paper: gradient checkpointing, ZeRO-2 with CPU-offloaded optimizer states, bf16, no torch.compile. Training time is the median over steps 10–30 and memory the peak per GPU; inference is one video on one GPU (50 steps, cfg 1, all streams decoded). With gradient checkpointing and the optimizer on the CPU, per-GPU memory is dominated by the model weights.

Under this configuration, peak GPU memory increases by 3.2 GB (10.2%), while training-step time and inference time increase by 2.7× and 3.9×, respectively. These measurements make the computational trade-off explicit alongside the improvement in RGB generation quality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers for their careful reading and constructive feedback. We appreciate their recognition of the idea of actuation-map representation (kohU, unEz) and the PHD dataset a useful resource (all reviewers). The main questions relate to the quantitative evaluation of generated actuation, the attribution of improvements to torque information, the coverage of fine-tuned baseline comparisons, the dataset and evaluation protocols, and computational cost.

To address these concerns, we provide additional actuation–motion consistency evaluations, matched fine-tuned baselines, and controlled ablations to better isolate the contribution of physical supervision. We further report reference-free evaluations, clarify the dataset construction and evaluation protocols, and quantify computational overhead. Notably, the original submission already includes a PHD-fine-tuned RGB-only baseline (Table 3); our new evaluations extend this comparison to physical plausibility. We have incorporated the reported results and their protocol details into the revised manuscript and appendix.

We provide detailed responses and supporting results below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.