- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
HandWM separates hand–object video prediction into explicit 3D interaction dynamics and geometry-conditioned observation synthesis, combining pixel-aligned surface conditioning, masked bidirectional geometry–appearance exchange, and optional restored supervision (Secs. 3.1–3.4). Evaluation with annotated future states improves DexYCB FVD from 31.9 to 16.9 and hand MPJPE from 31.1 to 21.5 mm, while two simulated trajectories demonstrate qualitative transfer.
Critical strengths
- The unified PAM comparison shows substantial improvements in both video quality and recovered hand geometry, with FVD decreasing from 31.9 to 16.9 and MPJPE from 31.1 to 21.5 mm while motion fidelity remains 0.834 (Table 1; Sec. 4.2, lines 314–323). Improvements on hand crops make the result relevant to the interaction region rather than solely to background appearance.
- The explicit state-to-image interface resolves projection and visibility before generation and supports swapping annotated states for simulated trajectories without retraining the observation model (Sec. 3.3, lines 220–247; Sec. 4.3, lines 461–468). This is a useful modular design for connecting 3D interaction systems to video synthesis, with Fig. 5 demonstrating both lifting and non-lifting outcomes.
Critical weaknesses
- The central world-modeling claim is not validated at the level of predicted interaction outcomes. The formulation predicts future observations from an initial state and hand controls, but every quantitative comparison supplies dataset-provided future hand–object states to the observation model (Eqs. (1)–(2); lines 483–484). These inputs already specify the object motion and occlusion that an interactive world model should predict. The FVD and MPJPE gains therefore establish better conditional synthesis, without establishing accurate state evolution or the benefit of factorizing dynamics from appearance in the complete pipeline. The two simulator examples explicitly test interface transfer rather than simulator accuracy (lines 461–468); reconstruction errors, fixed physical parameters, and the contact-adjustment procedure in Sec. 3.2 remain unevaluated as sources of forecast error. Moreover, even with annotated states, the reported metrics do not quantify object-state or contact-outcome fidelity: Appendix G shows an occluded-object retrieval failing while hand-only geometry metrics do not capture the full error (lines 1257–1265). The authors acknowledge these boundaries, but they leave the main interaction-prediction contribution materially less established than the observation-model contribution. Quantitative evaluation using reconstructed initial states and simulated futures, with object-motion or interaction-outcome measurements, is needed to substantiate the broader claim.
- The experiments do not isolate the added value of bidirectional geometry–appearance exchange from spatial masking and conditioning design. Table 2 replaces the proposed pathway with one-way full-field injection, simultaneously removing generation-to-control feedback and rendered-support gating (lines 368–375). Its FVD degradation from 16.9 to 98.2 supports the combined design, but does not establish whether feedback adds value beyond a simpler masked one-way pathway. The comparison with PAM also changes the conditioning representation and control architecture together. Table 4 does not resolve this attribution: downsampling the conditions measures changes in output sharpness, so a stronger response demonstrates sensitivity to condition detail, not more accurate use of contact or object geometry (lines 426–459). This matters because a central methodological contribution is how geometry and appearance interact, beyond providing a dense render and restricting control to its support. The paper is transparent that the ablation is joint, and the combined system's improvements remain valid. Matched masked one-way and unmasked bidirectional controls, followed by conditioning comparisons under a fixed control architecture, would establish which proposed mechanisms account for those gains.
Questions for authors
- [Clarification; Weakness 1] Beyond Fig. 5, are there existing quantitative results when annotated future states are replaced by the reconstructed-state/simulator pipeline? Please identify the available initial inputs, physical parameters, contact fitting, and optional feedforward assistance in those runs, and clarify which forecast claims are supported by the current evidence (Sec. 3.2; lines 461–468 and 483–484).
- [Clarification; Weakness 1] Do the existing generated videos or evaluation records provide object-pose/motion, contact, or interaction-outcome measurements that capture the failed retrieval in Appendix G? Please distinguish errors in the supplied 3D trajectory from failures to preserve that trajectory in the generated observations, and delimit what the present FVD/MF/hand-MPJPE results establish.
- [Clarification; Weakness 2] Are masked one-way control or bidirectional control without support gating already evaluated with identical PASC inputs, training settings, and capacity? Is there any existing comparison of PASC and alternative geometric inputs under the same control pathway? Please specify which independent mechanism conclusions the reported joint ablation supports.
- [Clarification; Weakness 2] For the paired condition-degradation study, are any existing measurements available that relate the sharpness response to hand/object geometry or interaction fidelity, rather than only Laplacian variance? Please clarify the intended evidential scope of this probe. The additional controls and full-pipeline measurements proposed above are directions for further validation; the rebuttal questions concern already available evidence and do not request a new experimental campaign.
8 comments
W1 & Q2. Addtional object metrics that capture the Appendix G failures
We further evaluated object alignment on the 1,600 DexYCB validation clips, using the annotated grasped object as the target. HandWM outperforms PAM on all six object metrics, indicating better object alignment during the hand-object interaction process. For further details, please refer to Appendix A in our revised manuscript.
Mask IoU quantifies agreement in the object's visible extent, while 2D and 3D center errors quantify its positional agreement across frames and thus how closely its trajectory follows the reference interaction.
| Method | Object mask IoU vs. annotations ↑ | Object 2D center error vs. annotations (px) ↓ | Object 3D center error vs. annotations (mm) ↓ | Object mask IoU vs. estimates on real videos ↑ | Object 2D center error vs. estimates on real videos (px) ↓ | Object 3D center error vs. estimates on real videos (mm) ↓ |
|---|---|---|---|---|---|---|
| PAM | 0.727 | 7.5 | 78.8 | 0.816 | 5.4 | 50.5 |
| HandWM | 0.759 | 6.2 | 76.8 | 0.859 | 3.9 | 41.7 |
We prompt SAM 2.1 once with the bounding box of the annotated first-frame mask and propagate the mask through the clip. Mask IoU measures the intersection over union with the reference mask. 2D center error is the Euclidean distance between the mask centroids in pixels. 3D center error is the Euclidean distance in millimeters between the estimated and reference object centers. To estimate the 3D center, we take the median of the ZoeDepth estimates inside the mask, calibrate the depth scale on the first frame, and back-project the mask centroid using the camera intrinsics.
The first three metrics use the annotated masks and posed object mesh centers. The last three use masks and centers obtained by running the same estimators on the real reference GT videos, which helps reduce systematic estimator differences.
W1 & Q2. Addtional analysis of the Appendix G failures
We further analyzed the 1,600 DexYCB validation clips by object occlusion and fingertip azimuth. Appendix J of our revised manuscript presents the full breakdown.
Comparison with PAM. Both methods exhibit similar trends across occlusion and azimuth groups, but HandWM consistently achieves better metrics in every group. HandWM’s performance advantage over PAM is more pronounced on challenging samples.
Object occlusion. As occlusion increases, object mask IoU generally declines, while object center errors increase. The training and validation sets have similar occlusion distributions, with about 71% and 68% of clips below 20% mean occlusion, respectively. Heavily occluded interactions are therefore less common during training, which may contribute to their difficulty. Severe occlusion is the main challenge in the scissors retrieval (Seq. 1088). The scissors are 92.9% occluded on average and fully hidden in about 80% of frames.
Highest-occlusion group (O8, 51.6% mean occlusion)
| Method | Hand LPIPS ↓ | Hand MPJPE (mm) ↓ | Object mask IoU ↑ | Object 2D center error (px) ↓ | Object 3D center error (mm) ↓ |
|---|---|---|---|---|---|
| PAM | 0.1683 | 33.5 | 0.676 | 13.4 | 74.3 |
| HandWM | 0.1304 | 23.4 | 0.754 | 8.4 | 63.4 |
Fingertip azimuth. Both methods have their highest hand MPJPE and lowest object IoU when the fingertips point away from the camera. A reach in this direction can place the hand or arm along the line of sight to the target and foreshorten the visible finger motion. The inverted wood-block grasp (Seq. 0602) has a mean azimuth of +16.7°, close to 0° (directly away). The viewing geometry makes the grasp difficult even when most of the object remains visible.
Fingertip azimuth group centered at 0° (pointing away from the camera)
| Method | Hand LPIPS ↓ | Hand MPJPE (mm) ↓ | Object mask IoU ↑ | Object 2D center error (px) ↓ | Object 3D center error (mm) ↓ |
|---|---|---|---|---|---|
| PAM | 0.2550 | 36.7 | 0.736 | 8.5 | 103.3 |
| HandWM | 0.1863 | 26.2 | 0.791 | 5.9 | 74.6 |
W2 & Q3. What do masked one-way control, ungated bidirectional control, and the PASC control signal contribute?
Thanks for the suggestion. We have added further ablation comparisons and analyses on the same 1,600 DexYCB s0 validation clips under the same training budget and evaluation protocol. These comparisons are designed to isolate the respective contributions of bidirectional feedback, support gating, and PASC. Together, the results demonstrate the advantages of our control modules and method.
1. Bidirectional feedback ablation
| Method | Full-frame LPIPS ↓ | Full-frame SSIM ↑ | Full-frame PSNR (dB) ↑ | Motion fidelity (MF) ↑ | FVD ↓ | Hand-crop FID ↓ | Hand-crop LPIPS ↓ | Hand-crop SSIM ↑ | Hand-crop PSNR (dB) ↑ |
|---|---|---|---|---|---|---|---|---|---|
| One-way PASC (gating on) | 0.1110 | 0.8755 | 26.63 | 0.8272 | 17.10 | 3.65 | 0.1601 | 0.7837 | 22.86 |
| Bidirectional PASC (gating on) | 0.1012 | 0.8822 | 27.13 | 0.8336 | 16.93 | 3.40 | 0.1351 | 0.8025 | 23.60 |
| Method | Hand MPJPE (mm) ↓ | Object mask IoU vs. annotations ↑ | Object 2D center error vs. annotations (px) ↓ | Object 3D center error vs. annotations (mm) ↓ | Object mask IoU vs. estimates on real videos ↑ | Object 2D center error vs. estimates on real videos (px) ↓ | Object 3D center error vs. estimates on real videos (mm) ↓ |
|---|---|---|---|---|---|---|---|
| One-way PASC (gating on) | 27.22 | 0.736 | 6.8 | 78.0 | 0.831 | 4.9 | 45.0 |
| Bidirectional PASC (gating on) | 21.53 | 0.759 | 6.2 | 76.8 | 0.859 | 3.9 | 41.7 |
Bidirectional feedback primarily improves hand–object control, while also enhancing visual texture fidelity. This supports the bidirectional interaction proposed in Sec. 3.3: the rendered PASC geometry and target RGB video are spatially aligned observations of the same underlying 3D state. The first update transfers appearance context from the generation stream to the control stream, and the second injects refined geometric control back into generation. This alignment makes the control signal easier to inject into the generation stream.
2. Support gating ablation
| Method | Full-frame LPIPS ↓ | Full-frame SSIM ↑ | Full-frame PSNR (dB) ↑ | Motion fidelity (MF) ↑ | FVD ↓ | Hand-crop FID ↓ | Hand-crop LPIPS ↓ | Hand-crop SSIM ↑ | Hand-crop PSNR (dB) ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Bidirectional PASC (gating off) | 0.1986 | 0.7856 | 21.80 | 0.7708 | 102.79 | 4.43 | 0.1917 | 0.7352 | 21.00 |
| Bidirectional PASC (gating on) | 0.1012 | 0.8822 | 27.13 | 0.8336 | 16.93 | 3.40 | 0.1351 | 0.8025 | 23.60 |
| Method | Hand MPJPE (mm) ↓ | Object mask IoU vs. annotations ↑ | Object 2D center error vs. annotations (px) ↓ | Object 3D center error vs. annotations (mm) ↓ | Object mask IoU vs. estimates on real videos ↑ | Object 2D center error vs. estimates on real videos (px) ↓ | Object 3D center error vs. estimates on real videos (mm) ↓ |
|---|---|---|---|---|---|---|---|
| Bidirectional PASC (gating off) | 23.54 | 0.755 | 6.6 | 90.8 | 0.847 | 4.4 | 60.2 |
| Bidirectional PASC (gating on) | 21.53 | 0.759 | 6.2 | 76.8 | 0.859 | 3.9 | 41.7 |
Support gating primarily improves visual fidelity and also strengthens hand–object control. In PASC renders, the black background marks locations outside the visible projected hand and object surfaces. These pixels carry no surface appearance information for the target view. With gating, the control branch injects features only at locations supported by visible hand and object surfaces, preventing the black padding from acting as a spurious appearance cue. Outside the mask, the two streams exchange no direct updates, leaving the generator to synthesize the background from its own features and the first-frame context.
Thank you for highlighting the gains over PAM and asking for the quantitative analysis of simulator-to-observation transfer, separate ablations of different modules and measurements of object fidelity.
W1 & Q1. Simulator-to-Observation Transfer Evaluation
For simulator-driven inputs, we test whether the unchanged observation model can follow hand–object trajectories produced by an external contact simulator rather than drawn from the dataset. Starting from the first-frame hand and object states of DexYCB sessions, the simulator replans grasps and updates object motion through contact dynamics. We evaluate 49 simulated trajectories, transforming each into all eight DexYCB camera views. For each view, we render PASC controls for HandWM and depth, segmentation, and hand-skeleton controls for the released PAM baseline. Each model generates two chained videos, with the second starting from the final generated frame of the first. This gives 784 trajectory–camera–video evaluations per model.
Because these replanned trajectories have no ground-truth RGB videos, we compare the generated observations with the simulated control states. The metrics measure how well each model follows the simulator-driven hand pose, object silhouette, and object motion. HandWM outperforms PAM on all four metrics, indicating more accurate alignment with the simulated hand–object interaction geometry. For additional visualizations, please see Fig. 18-19 in our revised manuscript.
For these runs, the available inputs comprised the first-frame RGB-D image, camera calibration, the ground-truth YCB object mesh and initial 6-DoF pose, and the initial MANO hand pose and shape parameters. Physical settings included object-specific YCB masses, mesh-derived inertias, gravity of 9.81 m/s², a sliding-friction coefficient of 1.0, a simulation timestep of 0.1 ms, a hand-contact reference time constant of 4 ms, and force-limited dynamic hand tracking with a 0.6 N·m finger-joint torque limit. Contact fitting used bounded pose adjustments with a 5 cm translation limit, a 45° rotation limit, an 8 cm contact-search distance, and a target fingertip-pad distance of −0.05 mm; a separate pregrasp fit used 2 cm clearance. Optional feedforward assistance was enabled in all runs: a static wrench-allocation solution supplied additional commands through the bounded hand actuators during holding and lifting. Within each success/failure pair, these settings and the initial state were identical. Object motion arose from simulated contact and gravity, without object-pose replay or externally applied object-guidance forces.
W1 & Q1. Benefits of Factorizing Dynamics and Observation Synthesis for World Modeling State Evolution
HandWM decouples interaction dynamics from observation synthesis. Given an initial hand–object state and a sequence of hand controls, the physics simulator (or a learned neural 3D dynamics model) resolves contact and gravity to predict how the object moves. The observation model then uses the predicted hand and object states to generate the video. This factorization makes the physical outcome explicit before any video frames are generated.
The following grasping example illustrates the importance of this distinction. Starting from the same state, two nearly identical hand trajectories differ slightly in finger placement. Only one establishes contact that can support the object during the lift. Because the upward hand motions look almost identical, a conventional video world model conditioned on hand motion alone could plausibly show the object lifting in both cases. HandWM’s simulator predicts a successful lift for one trajectory and a failed lift for the other, and the observation model preserves that distinction in the generated videos. For additional visualizations, please see Fig. 7 in our revised manuscript.
By assigning contact dynamics to the simulator, this factorization reduces the burden of learning physical state transitions from video directly and enables more precise prediction of interaction outcomes near the boundary between grasp success and failure.