acceptodds
Under review as a conference paper at ICLR 2027

HandWM: Hand–Object World Modeling with Explicit Interaction Dynamics

Abstract

Video world models aim to predict future visual observations conditioned on predefined controlling signals. While most existing methods focus on exploration (i.e., controlling the observation pose), the recent emergence of interactive world models begins to control the behavior of an agent in the video and model their interaction with the world. In this paper, we focus on hand–object interaction which poses a particular challenge because future observations depend on both 3D interaction dynamics and complex visual appearance. Direct action-conditioned video generation couples dynamics and appearance, leading to inaccurate interaction predictions. We propose a factorized interactive world modeling framework that disentangles explicit 3D interaction dynamics from visual observation synthesis. Given an initial observation and future hand controls, we first predict the 3D object geometry and the initial 3D hand pose and then leverage a simulator-based interaction module to produce an explicit 3D hand–object trajectory. We introduce pixel-aligned surface conditioning to rasterize future hand–object surfaces into the target view. The resulting maps provide pixel-aligned geometric guidance with explicit visibility and occlusion information. We further propose a bidirectional geometry–appearance interaction module to enable spatially aligned control and generation features to exchange information during synthesis. On DexYCB, our model reduces FVD from 31.9 to 16.9 and hand-pose MPJPE from 31.1 to 21.5 mm relative to the state of the art while preserving motion fidelity.

Primary area:
generative models
Keywords:
Hand-Object Interaction, HOI Video Generation, Sim-to-Real, Controllable Video Generation, Fine-Grained Generation, Synthetic Data, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sMvjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HandWM separates hand–object video prediction into explicit 3D interaction dynamics and geometry-conditioned observation synthesis, combining pixel-aligned surface conditioning, masked bidirectional geometry–appearance exchange, and optional restored supervision (Secs. 3.1–3.4). Evaluation with annotated future states improves DexYCB FVD from 31.9 to 16.9 and hand MPJPE from 31.1 to 21.5 mm, while two simulated trajectories demonstrate qualitative transfer.

Critical strengths
  1. The unified PAM comparison shows substantial improvements in both video quality and recovered hand geometry, with FVD decreasing from 31.9 to 16.9 and MPJPE from 31.1 to 21.5 mm while motion fidelity remains 0.834 (Table 1; Sec. 4.2, lines 314–323). Improvements on hand crops make the result relevant to the interaction region rather than solely to background appearance.
  2. The explicit state-to-image interface resolves projection and visibility before generation and supports swapping annotated states for simulated trajectories without retraining the observation model (Sec. 3.3, lines 220–247; Sec. 4.3, lines 461–468). This is a useful modular design for connecting 3D interaction systems to video synthesis, with Fig. 5 demonstrating both lifting and non-lifting outcomes.
Critical weaknesses
  1. The central world-modeling claim is not validated at the level of predicted interaction outcomes. The formulation predicts future observations from an initial state and hand controls, but every quantitative comparison supplies dataset-provided future hand–object states to the observation model (Eqs. (1)–(2); lines 483–484). These inputs already specify the object motion and occlusion that an interactive world model should predict. The FVD and MPJPE gains therefore establish better conditional synthesis, without establishing accurate state evolution or the benefit of factorizing dynamics from appearance in the complete pipeline. The two simulator examples explicitly test interface transfer rather than simulator accuracy (lines 461–468); reconstruction errors, fixed physical parameters, and the contact-adjustment procedure in Sec. 3.2 remain unevaluated as sources of forecast error. Moreover, even with annotated states, the reported metrics do not quantify object-state or contact-outcome fidelity: Appendix G shows an occluded-object retrieval failing while hand-only geometry metrics do not capture the full error (lines 1257–1265). The authors acknowledge these boundaries, but they leave the main interaction-prediction contribution materially less established than the observation-model contribution. Quantitative evaluation using reconstructed initial states and simulated futures, with object-motion or interaction-outcome measurements, is needed to substantiate the broader claim.
  2. The experiments do not isolate the added value of bidirectional geometry–appearance exchange from spatial masking and conditioning design. Table 2 replaces the proposed pathway with one-way full-field injection, simultaneously removing generation-to-control feedback and rendered-support gating (lines 368–375). Its FVD degradation from 16.9 to 98.2 supports the combined design, but does not establish whether feedback adds value beyond a simpler masked one-way pathway. The comparison with PAM also changes the conditioning representation and control architecture together. Table 4 does not resolve this attribution: downsampling the conditions measures changes in output sharpness, so a stronger response demonstrates sensitivity to condition detail, not more accurate use of contact or object geometry (lines 426–459). This matters because a central methodological contribution is how geometry and appearance interact, beyond providing a dense render and restricting control to its support. The paper is transparent that the ablation is joint, and the combined system's improvements remain valid. Matched masked one-way and unmasked bidirectional controls, followed by conditioning comparisons under a fixed control architecture, would establish which proposed mechanisms account for those gains.
Questions for authors
  1. [Clarification; Weakness 1] Beyond Fig. 5, are there existing quantitative results when annotated future states are replaced by the reconstructed-state/simulator pipeline? Please identify the available initial inputs, physical parameters, contact fitting, and optional feedforward assistance in those runs, and clarify which forecast claims are supported by the current evidence (Sec. 3.2; lines 461–468 and 483–484).
  2. [Clarification; Weakness 1] Do the existing generated videos or evaluation records provide object-pose/motion, contact, or interaction-outcome measurements that capture the failed retrieval in Appendix G? Please distinguish errors in the supplied 3D trajectory from failures to preserve that trajectory in the generated observations, and delimit what the present FVD/MF/hand-MPJPE results establish.
  3. [Clarification; Weakness 2] Are masked one-way control or bidirectional control without support gating already evaluated with identical PASC inputs, training settings, and capacity? Is there any existing comparison of PASC and alternative geometric inputs under the same control pathway? Please specify which independent mechanism conclusions the reported joint ablation supports.
  4. [Clarification; Weakness 2] For the paired condition-degradation study, are any existing measurements available that relate the sharpness response to hand/object geometry or interaction fidelity, rather than only Laplacian variance? Please clarify the intended evidential scope of this probe. The additional controls and full-pipeline measurements proposed above are directions for further validation; the rebuttal questions concern already available evidence and do not request a new experimental campaign.
8 comments

Thank you for highlighting the gains over PAM and asking for the quantitative analysis of simulator-to-observation transfer, separate ablations of different modules and measurements of object fidelity.

 

W1 & Q1. Simulator-to-Observation Transfer Evaluation

For simulator-driven inputs, we test whether the unchanged observation model can follow hand–object trajectories produced by an external contact simulator rather than drawn from the dataset. Starting from the first-frame hand and object states of DexYCB sessions, the simulator replans grasps and updates object motion through contact dynamics. We evaluate 49 simulated trajectories, transforming each into all eight DexYCB camera views. For each view, we render PASC controls for HandWM and depth, segmentation, and hand-skeleton controls for the released PAM baseline. Each model generates two chained videos, with the second starting from the final generated frame of the first. This gives 784 trajectory–camera–video evaluations per model.

 

Because these replanned trajectories have no ground-truth RGB videos, we compare the generated observations with the simulated control states. The metrics measure how well each model follows the simulator-driven hand pose, object silhouette, and object motion. HandWM outperforms PAM on all four metrics, indicating more accurate alignment with the simulated hand–object interaction geometry. For additional visualizations, please see Fig. 18-19 in our revised manuscript.

MethodWrist-aligned hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM39.30.6627.195
HandWM24.60.7224.985

 

For these runs, the available inputs comprised the first-frame RGB-D image, camera calibration, the ground-truth YCB object mesh and initial 6-DoF pose, and the initial MANO hand pose and shape parameters. Physical settings included object-specific YCB masses, mesh-derived inertias, gravity of 9.81 m/s², a sliding-friction coefficient of 1.0, a simulation timestep of 0.1 ms, a hand-contact reference time constant of 4 ms, and force-limited dynamic hand tracking with a 0.6 N·m finger-joint torque limit. Contact fitting used bounded pose adjustments with a 5 cm translation limit, a 45° rotation limit, an 8 cm contact-search distance, and a target fingertip-pad distance of −0.05 mm; a separate pregrasp fit used 2 cm clearance. Optional feedforward assistance was enabled in all runs: a static wrench-allocation solution supplied additional commands through the bounded hand actuators during holding and lifting. Within each success/failure pair, these settings and the initial state were identical. Object motion arose from simulated contact and gravity, without object-pose replay or externally applied object-guidance forces.

 

W1 & Q1. Benefits of Factorizing Dynamics and Observation Synthesis for World Modeling State Evolution

HandWM decouples interaction dynamics from observation synthesis. Given an initial hand–object state and a sequence of hand controls, the physics simulator (or a learned neural 3D dynamics model) resolves contact and gravity to predict how the object moves. The observation model then uses the predicted hand and object states to generate the video. This factorization makes the physical outcome explicit before any video frames are generated.

The following grasping example illustrates the importance of this distinction. Starting from the same state, two nearly identical hand trajectories differ slightly in finger placement. Only one establishes contact that can support the object during the lift. Because the upward hand motions look almost identical, a conventional video world model conditioned on hand motion alone could plausibly show the object lifting in both cases. HandWM’s simulator predicts a successful lift for one trajectory and a failed lift for the other, and the observation model preserves that distinction in the generated videos. For additional visualizations, please see Fig. 7 in our revised manuscript.

By assigning contact dynamics to the simulator, this factorization reduces the burden of learning physical state transitions from video directly and enables more precise prediction of interaction outcomes near the boundary between grasp success and failure.

W1 & Q2. Addtional object metrics that capture the Appendix G failures

We further evaluated object alignment on the 1,600 DexYCB validation clips, using the annotated grasped object as the target. HandWM outperforms PAM on all six object metrics, indicating better object alignment during the hand-object interaction process. For further details, please refer to Appendix A in our revised manuscript.

Mask IoU quantifies agreement in the object's visible extent, while 2D and 3D center errors quantify its positional agreement across frames and thus how closely its trajectory follows the reference interaction.

 

MethodObject mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
PAM0.7277.578.80.8165.450.5
HandWM0.7596.276.80.8593.941.7

 

We prompt SAM 2.1 once with the bounding box of the annotated first-frame mask and propagate the mask through the clip. Mask IoU measures the intersection over union with the reference mask. 2D center error is the Euclidean distance between the mask centroids in pixels. 3D center error is the Euclidean distance in millimeters between the estimated and reference object centers. To estimate the 3D center, we take the median of the ZoeDepth estimates inside the mask, calibrate the depth scale on the first frame, and back-project the mask centroid using the camera intrinsics.

The first three metrics use the annotated masks and posed object mesh centers. The last three use masks and centers obtained by running the same estimators on the real reference GT videos, which helps reduce systematic estimator differences.

 

W1 & Q2. Addtional analysis of the Appendix G failures

We further analyzed the 1,600 DexYCB validation clips by object occlusion and fingertip azimuth. Appendix J of our revised manuscript presents the full breakdown.

Comparison with PAM. Both methods exhibit similar trends across occlusion and azimuth groups, but HandWM consistently achieves better metrics in every group. HandWM’s performance advantage over PAM is more pronounced on challenging samples.

 

Object occlusion. As occlusion increases, object mask IoU generally declines, while object center errors increase. The training and validation sets have similar occlusion distributions, with about 71% and 68% of clips below 20% mean occlusion, respectively. Heavily occluded interactions are therefore less common during training, which may contribute to their difficulty. Severe occlusion is the main challenge in the scissors retrieval (Seq. 1088). The scissors are 92.9% occluded on average and fully hidden in about 80% of frames.

Highest-occlusion group (O8, 51.6% mean occlusion)

MethodHand LPIPS ↓Hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM0.168333.50.67613.474.3
HandWM0.130423.40.7548.463.4

 

Fingertip azimuth. Both methods have their highest hand MPJPE and lowest object IoU when the fingertips point away from the camera. A reach in this direction can place the hand or arm along the line of sight to the target and foreshorten the visible finger motion. The inverted wood-block grasp (Seq. 0602) has a mean azimuth of +16.7°, close to 0° (directly away). The viewing geometry makes the grasp difficult even when most of the object remains visible.

Fingertip azimuth group centered at 0° (pointing away from the camera)

MethodHand LPIPS ↓Hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM0.255036.70.7368.5103.3
HandWM0.186326.20.7915.974.6

W2 & Q3. What do masked one-way control, ungated bidirectional control, and the PASC control signal contribute?

Thanks for the suggestion. We have added further ablation comparisons and analyses on the same 1,600 DexYCB s0 validation clips under the same training budget and evaluation protocol. These comparisons are designed to isolate the respective contributions of bidirectional feedback, support gating, and PASC. Together, the results demonstrate the advantages of our control modules and method.

 

1. Bidirectional feedback ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
One-way PASC (gating on)0.11100.875526.630.827217.103.650.16010.783722.86
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
One-way PASC (gating on)27.220.7366.878.00.8314.945.0
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Bidirectional feedback primarily improves hand–object control, while also enhancing visual texture fidelity. This supports the bidirectional interaction proposed in Sec. 3.3: the rendered PASC geometry and target RGB video are spatially aligned observations of the same underlying 3D state. The first update transfers appearance context from the generation stream to the control stream, and the second injects refined geometric control back into generation. This alignment makes the control signal easier to inject into the generation stream.

 

2. Support gating ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
Bidirectional PASC (gating off)0.19860.785621.800.7708102.794.430.19170.735221.00
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
Bidirectional PASC (gating off)23.540.7556.690.80.8474.460.2
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Support gating primarily improves visual fidelity and also strengthens hand–object control. In PASC renders, the black background marks locations outside the visible projected hand and object surfaces. These pixels carry no surface appearance information for the target view. With gating, the control branch injects features only at locations supported by visible hand and object surfaces, preventing the black padding from acting as a spurious appearance cue. Outside the mask, the two streams exchange no direct updates, leaving the generator to synthesize the background from its own features and the first-frame context.

Reviewer_EoTKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HandWM factorizes hand–object video generation into explicit state evolution and observation synthesis. Future hand–object surfaces are rasterized through Pixel-Aligned Surface Conditioning (PASC), and a CogVideoX-based observation model couples control and generation features through bidirectional, spatially gated interaction. The paper also studies motion-associated observation degradation and selective restoration of training targets.

Under a matched DexYCB s0 protocol, HandWM reports FVD of 16.9 versus 31.9 for PAM and hand MPJPE of 21.5 versus 31.1 mm. Hand-crop FID improves from 3.81 to 3.14, while MF remains 0.834 (Table 1, lower block). These experiments evaluate observation synthesis from annotated future states. Simulator-driven transfer is illustrated by two qualitative examples in Figure 5. The evidence therefore supports improved conditional rendering more directly than the complete dynamics-to-video pipeline.

Critical strengths

The matched PAM comparison reports gains across multiple metrics. FVD decreases from 31.9 to 16.9, hand MPJPE decreases from 31.1 to 21.5 mm, and hand-crop FID improves from 3.81 to 3.14, while MF remains 0.834 (Table 1, lower block). The main model uses original training targets, so these gains are not explained by evaluation against restored references.

The paper reports useful negative results and identifiable failures. Motion-aware restoration improves MPJPE only from 21.5 to 21.0 mm while FVD worsens from 16.9 to 17.5 (Table 2); the authors acknowledge this limited benefit. Appendix G documents an atypical grasp with severe hand drift and a failed retrieval of an occluded object. These cases help expose limitations that average visual-quality scores do not capture.

Critical weaknesses

The experimental depth does not match the breadth of the contribution. Quantitative results use annotated future states, while simulator transfer consists of two qualitative examples and explicitly does not evaluate dynamics accuracy (p. 9; Figure 5). This leaves the state-prediction part of the world-model formulation unevaluated. Even within observation synthesis, the large FVD change from 16.9 to 98.2 in Table 2 removes feedback and spatial gating simultaneously, so it does not establish the independent benefit of bidirectional interaction. The restoration branch adds only a 0.5 mm pose improvement with worse FVD. Table 4 measures sensitivity through output sharpness, not geometric accuracy or successful interaction. The results demonstrate a promising configured system, but leave its principal source of improvement insufficiently explained relative to geometry-conditioned methods such as PAM.

The reported averages require a reliable account of measurement. The supplied evaluation scripts appear inconsistent with Appendix A and with standard LPIPS usage. In PAM/testing/evaluate_metrics_dexycb.py:71–80 and PAM/testing/evaluate_metrics_dexycb_hand.py:115–124, [0,1] inputs are passed to LPIPS without normalize=True. The MF path in evaluate_metrics_dexycb.py:93–119 instantiates SpaTracker rather than the CoTracker3 specified in Appendix A (p. 14), and appears to multiply already unnormalized video pixels by 255 before a tracker that divides by 255 internally. These discrepancies require confirmation against the exact scripts used for the tables and, if necessary, recomputation of the affected metrics on the same saved videos.

The failure analysis does not yet quantify interaction reliability. In Appendix G, the wood-block example approaches from the wrong side and displaces the object without grasping it; at frame 26, MPJPE is 147.3 mm versus 53.0 mm for PAM. In the scissors example, the occluded target never emerges. The latter error is not adequately described by hand MPJPE. The paper acknowledges these cases but does not establish their frequency or how performance changes with occlusion and atypical grasps. Figure 3 is explicitly selected on validation sequences where PAM has large hand-pose errors (p. 7), so it cannot indicate typical-case behavior. These gaps limit confidence that the mean improvements translate into reliable interaction generation.

Questions for authors

Metric clarification: Which exact scripts, tracker, LPIPS version, and normalization settings produced the reported MF and LPIPS values? Please reconcile the implementation paths above with Appendix A and, if necessary, recompute the affected metrics on the same saved PAM and HandWM videos.

Performance attribution: What happens when feedback and support gating are varied separately under the same PASC inputs and training budget? A gated one-way variant and an ungated bidirectional variant would clarify the large joint-ablation effect. How are the average gains distributed across validation sequences, especially those with occluded objects or atypical grasps?

Failure and scope clarification: For simulator-driven inputs, how well do generated object motion and interaction outcomes follow the specified states? Please distinguish errors from the simulator from errors of the observation model, and clarify what evidence extends beyond the two examples in Figure 5. For the Appendix G failures, what measures capture incorrect contact or unrecovered object state that hand MPJPE misses?

5 comments

Thank you for highlighting the gains over PAM and asking for the clarification of evaluation protocol, quantitative analysis of simulator-to-observation transfer, separate ablations of different modules and measurements of object fidelity.

 

W1 & Q2. What do masked one-way control, ungated bidirectional control, and the PASC control signal contribute?

Thanks for the suggestion. We have added further ablation comparisons and analyses on the same 1,600 DexYCB s0 validation clips under the same training budget and evaluation protocol. These comparisons are designed to isolate the respective contributions of bidirectional feedback, support gating, and PASC. Together, the results demonstrate the advantages of our control modules and method.

 

1. Bidirectional feedback ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
One-way PASC (gating on)0.11100.875526.630.827217.103.650.16010.783722.86
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
One-way PASC (gating on)27.220.7366.878.00.8314.945.0
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Bidirectional feedback primarily improves hand–object control, while also enhancing visual texture fidelity. This supports the bidirectional interaction proposed in Sec. 3.3: the rendered PASC geometry and target RGB video are spatially aligned observations of the same underlying 3D state. The first update transfers appearance context from the generation stream to the control stream, and the second injects refined geometric control back into generation. This alignment makes the control signal easier to inject into the generation stream.

 

2. Support gating ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
Bidirectional PASC (gating off)0.19860.785621.800.7708102.794.430.19170.735221.00
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
Bidirectional PASC (gating off)23.540.7556.690.80.8474.460.2
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Support gating primarily improves visual fidelity and also strengthens hand–object control. In PASC renders, the black background marks locations outside the visible projected hand and object surfaces. These pixels carry no surface appearance information for the target view. With gating, the control branch injects features only at locations supported by visible hand and object surfaces, preventing the black padding from acting as a spurious appearance cue. Outside the mask, the two streams exchange no direct updates, leaving the generator to synthesize the background from its own features and the first-frame context.

3. PASC versus PAM (geometry-conditioned method)

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
PAM (native one-way conditioning)0.12940.860326.320.834732.244.810.17050.770622.57
One-way PASC (gating off)0.18860.799022.450.754698.224.190.20070.734021.13
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
PAM (native one-way conditioning)31.080.7277.578.80.8165.450.5
One-way PASC (gating off)28.990.7446.990.40.8275.259.4

Both methods use one-way control without our rendered-support gate. PAM uses three geometry control signals—depth, segmentation, and hand skeleton—whereas our PASC rasterizes the 3D hand and object surfaces into a single target-aligned RGB stream, explicitly resolving projection, visibility, and occlusion. On hand–object control metrics, PASC outperforms PAM in most cases. While PASC’s weaker appearance scores are consistent with ungated injection treating black RGB padding as a texture cue. These results suggest that PASC provides a more effective control signal for video generation, particularly for hand–object spatial alignment.

 

W1 & Q3. Simulator-to-Observation Transfer Evaluation

For simulator-driven inputs, we test whether the unchanged observation model can follow hand–object trajectories produced by an external contact simulator rather than drawn from the dataset. Starting from the first-frame hand and object states of DexYCB sessions, the simulator replans grasps and updates object motion through contact dynamics. We evaluate 49 simulated trajectories, transforming each into all eight DexYCB camera views. For each view, we render PASC controls for HandWM and depth, segmentation, and hand-skeleton controls for the released PAM baseline. Each model generates two chained videos, with the second starting from the final generated frame of the first. This gives 784 trajectory–camera–video evaluations per model.

 

Because these replanned trajectories have no ground-truth RGB videos, we compare the generated observations with the simulated control states. The metrics measure how well each model follows the simulator-driven hand pose, object silhouette, and object motion. HandWM outperforms PAM on all four metrics, indicating more accurate alignment with the simulated hand–object interaction geometry. For additional visualizations, please see Fig. 18-19 in our revised manuscript.

MethodWrist-aligned hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM39.30.6627.195
HandWM24.60.7224.985

 

W1 & Q3. World Modeling State Prediction Evaluation

HandWM decouples interaction dynamics from observation synthesis. Given an initial hand–object state and a sequence of hand controls, the physics simulator (or a learned neural 3D dynamics model) resolves contact and gravity to predict how the object moves. The observation model then uses the predicted hand and object states to generate the video. This factorization makes the physical outcome explicit before any video frames are generated.

The following grasping example illustrates the importance of this distinction. Starting from the same state, two nearly identical hand trajectories differ slightly in finger placement. Only one establishes contact that can support the object during the lift. Because the upward hand motions look almost identical, a conventional video world model conditioned on hand motion alone could plausibly show the object lifting in both cases. HandWM’s simulator predicts a successful lift for one trajectory and a failed lift for the other, and the observation model preserves that distinction in the generated videos. For additional visualizations, please see Fig. 7 in our revised manuscript.

By assigning contact dynamics to the simulator, this factorization reduces the burden of learning physical state transitions from video directly and enables more precise prediction of interaction outcomes near the boundary between grasp success and failure.

W2 & Q1. LPIPS and motion fidelity recomputation

Thank you for identifying the evaluation errors, including the tracker mismatch and pixel-scaling discrepancy. We have corrected the evaluation scripts and updated the reported scores and protocol description accordingly. HandWM retains its LPIPS gains over PAM, while the recomputed MF scores are close.

MethodFull-frame LPIPS ↓Hand-crop LPIPS ↓MF ↑
PAM0.12940.17050.8347
HandWM0.10120.13510.8336

 

W3 & Q2–Q3. How do the gains vary with occlusion and grasp viewpoint?

We further analyzed the 1,600 DexYCB validation clips by object occlusion and fingertip azimuth. Appendix J of our revised manuscript presents the full breakdown.

Comparison with PAM. Both methods exhibit similar trends across occlusion and azimuth groups, but HandWM consistently achieves better metrics in every group. HandWM’s performance advantage over PAM is more pronounced on challenging samples.

 

Object occlusion. As occlusion increases, object mask IoU generally declines, while object center errors increase. The training and validation sets have similar occlusion distributions, with about 71% and 68% of clips below 20% mean occlusion, respectively. Heavily occluded interactions are therefore less common during training, which may contribute to their difficulty. Severe occlusion is the main challenge in the scissors retrieval (Seq. 1088). The scissors are 92.9% occluded on average and fully hidden in about 80% of frames.

Highest-occlusion group (O8, 51.6% mean occlusion)

MethodHand LPIPS ↓Hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM0.168333.50.67613.474.3
HandWM0.130423.40.7548.463.4

 

Fingertip azimuth. Both methods have their highest hand MPJPE and lowest object IoU when the fingertips point away from the camera. A reach in this direction can place the hand or arm along the line of sight to the target and foreshorten the visible finger motion. The inverted wood-block grasp (Seq. 0602) has a mean azimuth of +16.7°, close to 0° (directly away). The viewing geometry makes the grasp difficult even when most of the object remains visible.

Fingertip azimuth group centered at 0° (pointing away from the camera)

MethodHand LPIPS ↓Hand MPJPE (mm) ↓Object mask IoU ↑Object 2D center error (px) ↓Object 3D center error (mm) ↓
PAM0.255036.70.7368.5103.3
HandWM0.186326.20.7915.974.6

 

W3 & Q3. Addtional object metrics that capture the Appendix G failures

We evaluated object alignment on the 1,600 DexYCB validation clips, using the annotated grasped object as the target. HandWM outperforms PAM on all six object metrics, indicating better object alignment during the hand-object interaction process. For further details, please refer to Appendix A in our revised manuscript.

Mask IoU quantifies agreement in the object's visible extent, while 2D and 3D center errors quantify its positional agreement across frames and thus how closely its trajectory follows the reference interaction. HandWM improves all six over PAM, indicating better object alignment during the hand-object interaction process.

 

MethodObject mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
PAM0.7277.578.80.8165.450.5
HandWM0.7596.276.80.8593.941.7

 

We prompt SAM 2.1 once with the bounding box of the annotated first-frame mask and propagate the mask through the clip. Mask IoU measures the intersection over union with the reference mask. 2D center error is the Euclidean distance between the mask centroids in pixels. 3D center error is the Euclidean distance in millimeters between the estimated and reference object centers. To estimate the 3D center, we take the median of the ZoeDepth estimates inside the mask, calibrate the depth scale on the first frame, and back-project the mask centroid using the camera intrinsics.

The first three metrics use the annotated masks and posed object mesh centers. The last three use masks and centers obtained by running the same estimators on the real reference GT videos, which helps reduce systematic estimator differences.

Reviewer_NZtfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HandWM separates hand--object video prediction into 3D state evolution and video synthesis. It reconstructs an initial interaction state, uses MuJoCo to evolve that state under hand controls, and renders the geometry as Pixel-Aligned Surface Conditioning (PASC). A CogVideoX-based observation model combines these renders with appearance features through masked bidirectional interaction. The main quantitative experiments use dataset-provided future states on DexYCB s0, where the method improves FVD from 31.9 to 16.9 and hand MPJPE from 31.1 to 21.5 mm over PAM. The paper also studies motion-related degradation in training images and shows two examples driven by simulated trajectories.

Critical strengths
  1. The improvement over PAM is substantial under the shared evaluation protocol. FVD falls from 31.9 to 16.9 and hand MPJPE from 31.1 to 21.5 mm, with improvements in full-frame LPIPS and hand-crop FID as well (Table 1). Better hand geometry alongside better video quality makes this a useful advance for interaction video synthesis.

  2. PASC provides a practical connection between explicit geometry and the video generator. Rendering the state into the target view gives the control branch a spatial correspondence with the output, while the simulator examples show that the same observation model can consume trajectories from another state source without retraining (Figure 5).

  3. The analysis of motion and supervision quality is useful beyond the particular architecture. The negative relationship between hand speed and image quality appears across DexYCB, EgoDex and OakInk2 (Table 3). The restoration experiment then shows how little of this problem is recovered by changing the targets alone: MPJPE improves slightly, while overall video fidelity does not. This helps locate a remaining bottleneck.

Critical weaknesses
  1. The visualizations of HandWM-generated interactions are all based on DexYCB. The paper does not show generated results for more complex HOI settings, such as bimanual manipulation or egocentric views. This leaves limited evidence of how well the method handles a wider range of actions, coordinated hand movements and viewpoints

  2. The one-way ablation removes both generation-to-control feedback and rendered-support gating. Its FVD of 98.2, compared with 16.9 for the full model, shows a large difference but leaves the source of that difference unresolved (Table 2). A masked one-way branch could help determine whether bidirectional feedback is needed or whether spatially restricting the control already accounts for most of the gain.

  3. The quantitative evaluation does not directly measure whether the object follows the intended interaction. Appendix G shows misplaced objects, failed grasps and occlusion failures that can coexist with a plausible hand trajectory. Hand MPJPE and global appearance metrics leave an important part of the claimed interaction fidelity unmeasured.

Questions for authors
  1. Could you provide more visualizations of HandWM-generated interactions on different datasets, such as EgoDex and OakInk2 mentioned in the paper, as well as TACO (Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding)? Examples of bimanual manipulation and egocentric views would help assess the method on more complex HOI scenarios.

  2. How does masked one-way injection perform under the same training budget? A comparison with unmasked bidirectional interaction would also help separate the effects of feedback and support gating.

  3. Can you quantify object pose or motion agreement and contact/grasp success on the generated videos? In particular, how often do the object failures in Appendix G occur relative to PAM?

6 comments

Thank you for highlighting the gains over PAM and asking for broader examples, separate ablations, and measurements of object fidelity.

 

W1 & Q1. How does HandWM perform on bimanual and egocentric interactions?

Thanks for the suggestion. We further evaluate our mthod on more complex HOI settings including on the OakInk2 dataset, which contains both exo- and ego-views with bimanual hand–object manipulation in complex tasks. We evaluated HandWM and PAM on 1,600 OakInk2 clips. HandWM outperforms PAM on nearly all reported metrics. For additional visualizations, please see Fig. 17 in our revised manuscript.

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
PAM0.07930.868326.110.992236.206.430.10090.800324.41
HandWM0.06180.882527.201.034231.867.360.08130.825624.88
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
PAM21.730.53531.883.00.8476.629.6
HandWM19.330.54631.383.00.8686.530.0

We have added OakInk2 experiments to complement the single-hand interactions shown on DexYCB. OakInk2 contains bimanual hand–object interactions, so the two datasets together cover both single-hand and bimanual manipulation. Both datasets provide multiple camera viewpoints, and OakInk2 also includes egocentric views. HandWM’s stronger performance on DexYCB and OakInk2 suggests that it remains robust across single- and bimanual interactions and multiple viewpoints, including egocentric views.

W2 & Q2. What do masked one-way control, ungated bidirectional control, and the PASC control signal contribute?

Thanks for the suggestion. We have added further ablation comparisons and analyses on the same 1,600 DexYCB s0 validation clips under the same training budget and evaluation protocol. These comparisons are designed to isolate the respective contributions of bidirectional feedback, support gating, and PASC. Together, the results demonstrate the advantages of our control modules and method.

 

1. Bidirectional feedback ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
One-way PASC (gating on)0.11100.875526.630.827217.103.650.16010.783722.86
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
One-way PASC (gating on)27.220.7366.878.00.8314.945.0
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Bidirectional feedback primarily improves hand–object control, while also enhancing visual texture fidelity. This supports the bidirectional interaction proposed in Sec. 3.3: the rendered PASC geometry and target RGB video are spatially aligned observations of the same underlying 3D state. The first update transfers appearance context from the generation stream to the control stream, and the second injects refined geometric control back into generation. This alignment makes the control signal easier to inject into the generation stream.

 

2. Support gating ablation

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
Bidirectional PASC (gating off)0.19860.785621.800.7708102.794.430.19170.735221.00
Bidirectional PASC (gating on)0.10120.882227.130.833616.933.400.13510.802523.60
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
Bidirectional PASC (gating off)23.540.7556.690.80.8474.460.2
Bidirectional PASC (gating on)21.530.7596.276.80.8593.941.7

Support gating primarily improves visual fidelity and also strengthens hand–object control. In PASC renders, the black background marks locations outside the visible projected hand and object surfaces. These pixels carry no surface appearance information for the target view. With gating, the control branch injects features only at locations supported by visible hand and object surfaces, preventing the black padding from acting as a spurious appearance cue. Outside the mask, the two streams exchange no direct updates, leaving the generator to synthesize the background from its own features and the first-frame context.

3. PASC versus PAM (geometry-conditioned method)

MethodFull-frame LPIPS ↓Full-frame SSIM ↑Full-frame PSNR (dB) ↑Motion fidelity (MF) ↑FVD ↓Hand-crop FID ↓Hand-crop LPIPS ↓Hand-crop SSIM ↑Hand-crop PSNR (dB) ↑
PAM (native one-way conditioning)0.12940.860326.320.834732.244.810.17050.770622.57
One-way PASC (gating off)0.18860.799022.450.754698.224.190.20070.734021.13
MethodHand MPJPE (mm) ↓Object mask IoU vs. annotations ↑Object 2D center error vs. annotations (px) ↓Object 3D center error vs. annotations (mm) ↓Object mask IoU vs. estimates on real videos ↑Object 2D center error vs. estimates on real videos (px) ↓Object 3D center error vs. estimates on real videos (mm) ↓
PAM (native one-way conditioning)31.080.7277.578.80.8165.450.5
One-way PASC (gating off)28.990.7446.990.40.8275.259.4

Both methods use one-way control without our rendered-support gate. PAM uses three geometry control signals—depth, segmentation, and hand skeleton—whereas our PASC rasterizes the 3D hand and object surfaces into a single target-aligned RGB stream, explicitly resolving projection, visibility, and occlusion. On hand–object control metrics, PASC outperforms PAM in most cases. While PASC’s weaker appearance scores are consistent with ungated injection treating black RGB padding as a texture cue. These results suggest that PASC provides a more effective control signal for video generation, particularly for hand–object spatial alignment.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. They recognized HandWM’s factorization of hand–object world modeling into explicit 3D interaction dynamics and observation synthesis, as well as its improvements over PAM in both video quality and hand-object geometry accuracy.

During the rebuttal, we added object-level metrics showing that HandWM follows reference object trajectories more accurately than PAM, matched ablations isolating the contributions of feedback and masking modules, and an OakInk2 evaluation extending the comparison to bimanual and egocentric interactions. These experiments further demonstrate HandWM’s advantages in preserving interaction geometry across settings. Two reviewers confirmed that the additions addressed their main concerns.

We appreciate the third reviewer’s follow-up and believe our additional experiments address the remaining concerns. The missed-grasp tests demonstrate an advantage of our factorization over conventional video world models in predicting interaction outcomes. HandWM preserves the distinction between a successful lift and a missed grasp, while Hand2World and EgoHOI can show objects lifting or remaining suspended even when the hand misses them. We also evaluated sensitivity to reconstruction errors in 49 paired grasp simulations and tested zero-shot transfer between DexYCB and OakInk2 to assess generalization to unfamiliar settings.

We hope the AC will consider these clarifications and improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.