- Low quality of the generated motions.
The overall quality of the generated results is relatively low. Many examples exhibit noticeable artifacts, including hand–object penetrations, floating objects, unstable grasps, and unnatural motions. These issues are particularly concerning because the demonstrated interactions are not very complex, with many examples involving relatively simple grasping and moving of objects such as cups.
- Unclear motivation and practical value of the forecasting task.
I find the motivation and practical purpose of the proposed forecasting task unclear. For example, in the supplementary videos, the future object trajectory and final pose seem largely arbitrary. Given a short observation of someone holding a cup, there is no clear indication of where the person intends to move it. Many different future motions could be equally plausible.
What is the purpose of predicting a particular future trajectory in such cases? Why should the generated motion be evaluated against the specific ground-truth future recorded in the video? What are the intended applications of this type of hand–object motion forecasting?
It seems more appropriate to formulate this as conditional generation of plausible hand–object motions rather than forecasting a particular future. The paper does not clearly explain what practical value the proposed forecasting formulation provides.
- Lack of physical grounding in the joint motion generation formulation.
The object does not move independently during manipulation; its motion is largely determined by hand–object contact and the forces applied to it. Therefore, I question the motivation for jointly generating hand and object trajectories without explicitly enforcing their physical relationship.
Would it be more appropriate to generate the hand motion and use a physics simulator or physics-constrained optimization to determine the resulting object motion? The proposed unified generative formulation appears to produce numerous physical inconsistencies, including the penetration and floating artifacts mentioned above. The paper does not sufficiently justify the advantages of its approach over more physically grounded alternatives.
- Unclear presentation and limited technical novelty.
Overall, the paper does not clearly communicate its central motivation or the specific problem it aims to solve. It combines several existing techniques, including masked motion modeling, flow matching, hand–object relation representations, and contact-based losses. However, I do not see a clear technical innovation in the individual components or their combination. The authors should better distinguish their contributions from existing hand–object motion generation approaches.
- Overly permissive penetration threshold in the data extraction pipeline.
The 20 mm penetration threshold used in the data extraction quality gate seems overly permissive for hand–object interactions. A penetration of this magnitude is already substantial relative to the size of a finger or fingernail and can result in clearly implausible grasp configurations. Given that accurate hand–object contact is central to the proposed method, stricter geometric validation seems necessary.
- Value and quality of the extracted dataset.
The large-scale hand–object interaction dataset reconstructed from internet videos is potentially a valuable contribution. However, the reconstructed data also appear to contain considerable artifacts, including inaccurate hand–object alignment, penetration, and implausible contact configurations.
The authors should consider more thorough data cleaning and geometric refinement to improve the reliability of the extracted trajectories. Without sufficiently high-quality contact and motion supervision, it is unclear how much additional value this dataset provides over existing annotated hand–object datasets, or whether it can support substantially better interaction generation than existing methods.
Thank you for the comments. We answer each point below.
What is new beyond masked completion. We cite both works in Section 2. Masked completion itself is not our contribution, and like CondMDI [1] we train with random masks. What is new is how we apply it to hand–object interaction with two key aspects: masks over a grid of hands, object and their relations, and a relation token that the model generates with the poses. CondMDI masks keyframes and joints of a single body and is guided by text, and HO-Flow [2] uses masking only to order generation from text and a canonical object, so it never completes observed motion. Neither observes part of a recorded hand–object interaction and has its output scored against the part it did not see, which is our setting.
The first aspect is how we mask. Each window is a grid with a row for each hand, the object and the two hand–object relations, and a column for each frame. Hiding whole rows enables prediction of the object from the hands or the hands from the object, and hiding time spans gives forecasting, motion toward a goal and infilling. One model therefore covers all of these tasks, and under the energy score [3], which scores the 20 samples together by averaging their errors and subtracting half their average distance to each other, it beats separately trained task specialists in 7 of the 12 task columns of Table 10, including object and hand forecasting, hands→object and object→hands.
The second aspect is the relation token. It holds each hand's wrist pose in the object's frame and the distance from each fingertip to the object surface. The model generates it together with the poses, so the hand–object relation is predicted directly rather than relying on it to emerge from independently generated poses, and it is hidden whenever either pose is hidden so it cannot reveal the answer. Without it, the share of grasp frames in which the fingertips come off the object rises from 24.9% to 28.1% (Table 9).
Input-aligned comparison. We ran the same trained model on the 331 windows of Table 2 under two additional settings: without hand observations, matching the inputs to ObjectForesight, and without object observations, matching the inputs to the hand forecasting methods. The numbers come from a re-run of Table 2 in which the hand forecasters start at the last observed wrist and our model uses estimated depth. FDE in mm, for the best of 20 samples as in Table 2 and for the energy score [3] over the 20 samples (lower is better):
Without hand observations our object FDE is still lower than ObjectForesight's under both measures, and observing the hands lowers it further. Without object observations our wrist FDE is lower than ForeHand4D's under both measures.
Goal specialist. The specialist trains on goal tasks only. In our model each partial goal, where only the hands' or only the object's final pose is given, is about 5% of the training windows, against 25% for the specialist. Scored with the energy score [3] over the 20 samples, the two models are close except where an entity's final pose is not given:
The goal task is the second hardest task, after forecasting, and the specialist saw it five times more often. We expect a larger share of partial goals in training, or longer training, to narrow this gap.
Contact at inference. You are right that Eq. 3 is only used in training. It still matters at inference, because every sample comes from the velocity field that it helped train. Removing Eq. 3 and the other physical losses gives 17% more detached grasp frames and 10% higher fingertip error during contact (Table 9).
References
[1] S. Cohan et al. Flexible motion in-betweening with diffusion models. In SIGGRAPH, 2024.
[2] Z. Chen et al. HO-Flow: Generalizable hand-object interaction generation with latent flow matching. In ECCV, 2026.
[3] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 102(477):359–378, 2007.