acceptodds
Withdrawn from ICLR 2027

Grasping the Future: Joint 3D Hand–Object Forecasting from Internet Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present Grasping-the-Future, a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model to forecast future interactions, predict hand motion given object motion or vice versa, generate both trajectories toward a desired final object pose, and complete partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets to learn future motion from partial observations. We demonstrate strong performance across all tasks, outperforming recent state-of-the-art models. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
hand-object interaction, 3D motion forecasting, human motion generation, flow matching, generative models, masked modeling, egocentric video, 4D reconstruction, learning from human videos, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cBsKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper unifies hand-object forecasting and conditional generation as masked spatiotemporal completion, trained with flow matching on trajectories from annotated datasets and internet videos. However, the masked completion formulation follows prior work closely, and the claimed benefits of unification and contact preservation are not convincingly supported.

Critical strengths

a. The observation mask alone switches the model between seven tasks, so one model replaces separate forecasting, cross-entity prediction, goal-conditioned, and infilling models.

b. The extraction pipeline chains pretrained perception models with joint optimization and automated quality gates, yielding over 62,000 trajectories from internet videos and providing a concrete path to scale hand-object supervision.

Critical weaknesses

a. Masked spatiotemporal completion for motion generation has been explored in Cohan et al., 2024 and Chen et al., 2026c. The paper applies it to a hand-object token grid but does not discuss what is conceptually new beyond this design choice.

b. Table 2 is not a fair comparison, since each method uses its own inputs. ObjectForesight is the closest baseline, differing from the proposed model only in observed hand states. The author should add an input-aligned comparison against it to separate the formulation's effect from this extra input.

c. Table 10 shows the goal specialist outperforms the unified model on all goal-conditioned tasks, 3 out of 7 tasks. The specific reason should be discussed.

d. Eq. 3 acts on decoded trajectories during training but never enters the sampling process, so nothing enforces contact at inference.

Questions for authors

see above

1 comment

Thank you for the comments. We answer each point below.

What is new beyond masked completion. We cite both works in Section 2. Masked completion itself is not our contribution, and like CondMDI [1] we train with random masks. What is new is how we apply it to hand–object interaction with two key aspects: masks over a grid of hands, object and their relations, and a relation token that the model generates with the poses. CondMDI masks keyframes and joints of a single body and is guided by text, and HO-Flow [2] uses masking only to order generation from text and a canonical object, so it never completes observed motion. Neither observes part of a recorded hand–object interaction and has its output scored against the part it did not see, which is our setting.

The first aspect is how we mask. Each window is a grid with a row for each hand, the object and the two hand–object relations, and a column for each frame. Hiding whole rows enables prediction of the object from the hands or the hands from the object, and hiding time spans gives forecasting, motion toward a goal and infilling. One model therefore covers all of these tasks, and under the energy score [3], which scores the 20 samples together by averaging their errors and subtracting half their average distance to each other, it beats separately trained task specialists in 7 of the 12 task columns of Table 10, including object and hand forecasting, hands→object and object→hands.

The second aspect is the relation token. It holds each hand's wrist pose in the object's frame and the distance from each fingertip to the object surface. The model generates it together with the poses, so the hand–object relation is predicted directly rather than relying on it to emerge from independently generated poses, and it is hidden whenever either pose is hidden so it cannot reveal the answer. Without it, the share of grasp frames in which the fingertips come off the object rises from 24.9% to 28.1% (Table 9).

Input-aligned comparison. We ran the same trained model on the 331 windows of Table 2 under two additional settings: without hand observations, matching the inputs to ObjectForesight, and without object observations, matching the inputs to the hand forecasting methods. The numbers come from a re-run of Table 2 in which the hand forecasters start at the last observed wrist and our model uses estimated depth. FDE in mm, for the best of 20 samples as in Table 2 and for the energy score [3] over the 20 samples (lower is better):

FDE (mm)Object, best of 20Object, energy scoreWrists L/R, best of 20Wrists L/R, energy score
ObjectForesight127.6130.4––
ForeHand4D––134.3/142.2122.8/133.6
Ours, no hand observations122.6102.6––
Ours, no object observations––131.5/119.6101.5/97.1
Ours105.791.1126.5/112.0103.4/92.4

Without hand observations our object FDE is still lower than ObjectForesight's under both measures, and observing the hands lowers it further. Without object observations our wrist FDE is lower than ForeHand4D's under both measures.

Goal specialist. The specialist trains on goal tasks only. In our model each partial goal, where only the hands' or only the object's final pose is given, is about 5% of the training windows, against 25% for the specialist. Scored with the energy score [3] over the 20 samples, the two models are close except where an entity's final pose is not given:

Energy score (mm)All given, objectAll given, handsHands given, objectHands given, handsObject given, objectObject given, hands
Goal specialist53.660.053.061.354.761.0
Ours51.059.371.162.053.570.5

The goal task is the second hardest task, after forecasting, and the specialist saw it five times more often. We expect a larger share of partial goals in training, or longer training, to narrow this gap.

Contact at inference. You are right that Eq. 3 is only used in training. It still matters at inference, because every sample comes from the velocity field that it helped train. Removing Eq. 3 and the other physical losses gives 17% more detached grasp frames and 10% higher fingertip error during contact (Table 9).

References

[1] S. Cohan et al. Flexible motion in-betweening with diffusion models. In SIGGRAPH, 2024.

[2] Z. Chen et al. HO-Flow: Generalizable hand-object interaction generation with latent flow matching. In ECCV, 2026.

[3] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 102(477):359–378, 2007.

Reviewer_CBZUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Grasping-the-Future, a unified generative model for 3D hand–object motion prediction. The model jointly generates hand wrist motion, finger articulation, and object trajectories using a flow-matching transformer with masked spatiotemporal modeling. By changing the observation masks, the same model supports multiple tasks, including future motion forecasting, hand-to-object and object-to-hand motion prediction, goal-conditioned generation, and temporal infilling. The method also incorporates explicit hand–object relation representations and contact-related losses to encourage physically plausible interactions.

To obtain large-scale 3D training data, the authors develop a pipeline that reconstructs hand–object trajectories from internet RGB videos using pretrained perception models and geometric optimization. They extract approximately 62K trajectories and combine them with existing annotated 3D hand–object datasets for training. Experiments evaluate trajectory prediction accuracy, contact quality, and the effect of the extracted training data. The authors also demonstrate robotic retargeting of the predicted motions in simulation.

Critical strengths

The main strength of this paper is the effort to extract large-scale 3D hand–object interaction data from real-world internet videos. The authors develop a pipeline combining pretrained perception models with geometric optimization and quality filtering, and recover over 62K hand–object motion trajectories. Given the scarcity of large-scale 3D hand–object interaction datasets, this is potentially a valuable contribution. The experiments also show that incorporating the extracted data improves prediction performance compared with training on existing annotated datasets alone. Although the reconstructed data still contain noticeable artifacts, the approach offers a promising way to scale up 3D hand–object interaction data collection.

Critical weaknesses
  1. Low quality of the generated motions.

The overall quality of the generated results is relatively low. Many examples exhibit noticeable artifacts, including hand–object penetrations, floating objects, unstable grasps, and unnatural motions. These issues are particularly concerning because the demonstrated interactions are not very complex, with many examples involving relatively simple grasping and moving of objects such as cups.

  1. Unclear motivation and practical value of the forecasting task.

I find the motivation and practical purpose of the proposed forecasting task unclear. For example, in the supplementary videos, the future object trajectory and final pose seem largely arbitrary. Given a short observation of someone holding a cup, there is no clear indication of where the person intends to move it. Many different future motions could be equally plausible.

What is the purpose of predicting a particular future trajectory in such cases? Why should the generated motion be evaluated against the specific ground-truth future recorded in the video? What are the intended applications of this type of hand–object motion forecasting?

It seems more appropriate to formulate this as conditional generation of plausible hand–object motions rather than forecasting a particular future. The paper does not clearly explain what practical value the proposed forecasting formulation provides.

  1. Lack of physical grounding in the joint motion generation formulation.

The object does not move independently during manipulation; its motion is largely determined by hand–object contact and the forces applied to it. Therefore, I question the motivation for jointly generating hand and object trajectories without explicitly enforcing their physical relationship.

Would it be more appropriate to generate the hand motion and use a physics simulator or physics-constrained optimization to determine the resulting object motion? The proposed unified generative formulation appears to produce numerous physical inconsistencies, including the penetration and floating artifacts mentioned above. The paper does not sufficiently justify the advantages of its approach over more physically grounded alternatives.

  1. Unclear presentation and limited technical novelty.

Overall, the paper does not clearly communicate its central motivation or the specific problem it aims to solve. It combines several existing techniques, including masked motion modeling, flow matching, hand–object relation representations, and contact-based losses. However, I do not see a clear technical innovation in the individual components or their combination. The authors should better distinguish their contributions from existing hand–object motion generation approaches.

  1. Overly permissive penetration threshold in the data extraction pipeline.

The 20 mm penetration threshold used in the data extraction quality gate seems overly permissive for hand–object interactions. A penetration of this magnitude is already substantial relative to the size of a finger or fingernail and can result in clearly implausible grasp configurations. Given that accurate hand–object contact is central to the proposed method, stricter geometric validation seems necessary.

  1. Value and quality of the extracted dataset.

The large-scale hand–object interaction dataset reconstructed from internet videos is potentially a valuable contribution. However, the reconstructed data also appear to contain considerable artifacts, including inaccurate hand–object alignment, penetration, and implausible contact configurations.

The authors should consider more thorough data cleaning and geometric refinement to improve the reliability of the extracted trajectories. Without sufficiently high-quality contact and motion supervision, it is unclear how much additional value this dataset provides over existing annotated hand–object datasets, or whether it can support substantially better interaction generation than existing methods.

Questions for authors

The paper compares it's best of 20 tries with other methods. Is it really a fair comparison?

2 comments

Thank you for the detailed review and for recognising the value of the reconstructed data. We address each concern below.

Value and quality of the data. We tested our extraction pipeline on datasets with ground-truth hands and objects and compared it with HOPformer. It is more accurate in every comparison below, even on EPIC-Contact, the dataset HOPformer is trained on:

Reconstruction error (mm)HOPformerOurs
EPIC-Contact hand, HOPformer given hand crops and meshes (Table 7)19.716.0
EPIC-Contact hand / object, both from the video alone23.9 / 24.015.8 / 18.3
DexYCB hand / object, third-person view96.6 / 28.010.3 / 20.7

Training with our data improves performance despite its artifacts (Table 1). With it, the energy score (defined below) drops in all 12 task columns of the annotated windows, which do not come from our pipeline, and on internet video a single forecast lands 19–30% closer to what happened. We agree that further cleaning would help, and we will release every trajectory with its quality-check statistics so that users can select stricter subsets.

Unclear motivation and practical value of the forecasting task. Future prediction and world modeling are essential for learning manipulation from human videos. To plan effectively, a robot needs to anticipate the outcomes of its actions, which motivates our forecasting task. Regarding the comment on conditional generation, six of our seven tasks involve conditional generation. For example, given the final pose of an object, our model generates plausible hand motions that could achieve that pose. As for evaluating against the recorded future, the cup could indeed go to many places, which is why the model draws 20 samples, and the recorded future is one of them. The energy score (defined below) is strictly proper, so its expected value is lowest only when the samples follow the distribution of real futures, not when they match one recording.

Lack of physical grounding in the joint motion generation formulation. Using a simulator to refine the generated motions is a great idea. However, accurately simulating object motion from generated hand poses would require knowledge of the object's mass, inertia, and friction, as well as millimeter-level contact accuracy. These properties are not available from monocular internet videos, and even small errors in the generated hand poses could cause the simulated object to fall. We incorporate physics after forecasting in Section 4.3, where we use SPIDER to retarget our predicted motions to simulated robot hands. Even the reference motions, mostly from motion capture, could be reproduced in simulation for only 103 of 170 recordings, and on those SPIDER reproduces 73.8% of our forecasts.

Unclear presentation and limited technical novelty. We predict how both hands and an object move in 3D from part of a real interaction, whether that part is its first half second, the motion of one entity, a goal pose or the two ends of a gap, for robot manipulation as described above. The hand–object forecasting we know of, TACO [1] and HOIGPT [2], is trained and tested on motion capture only, and each released forecaster in Table 2 predicts either the hands or the object. Generation methods start from text, a grasp or a goal pose without an observed interaction, while each of our tasks continues or completes one.

Masked modelling, flow matching and contact losses are established tools, and we do not claim them as new. Our contribution is how we combine them for this problem. Masking whole hands or the object as well as time spans is what lets one model handle all of these tasks. Object-frame wrist poses and fingertip distances appear in prior work [3, 4, 5] as inputs or losses. We instead generate them as a relation token alongside the poses and hide it whenever either pose is hidden, so the model must predict the relation rather than read it. Removing this token raises detached grasp frames from 24.9% to 28.1% (Table 9). Additionally, a unified model for all these tasks is unique and novel. Finally, as you kindly noted among the strengths, the 62K trajectories we reconstruct from internet video are a contribution of their own, and they improve the model even on the annotated windows, as shown above.

References

[1] Y. Liu et al. TACO: Benchmarking generalizable bimanual tool-action-object understanding. In CVPR, 2024.

[2] M. Huang et al. HOIGPT: Learning long-sequence hand-object interaction with language models. In CVPR, 2025.

[3] H. Zhang et al. ManipNet: Neural manipulation synthesis with a hand-object spatial representation. ACM TOG, 40(4), 2021.

[4] O. Taheri et al. GOAL: Generating 4D whole-body motion for hand-object grasping. In CVPR, 2022.

[5] K. Zhou et al. TOCH: Spatio-temporal object-to-hand correspondence for motion refinement. In ECCV, 2022.

Overly permissive penetration threshold in the data extraction pipeline. The 20 mm value is not a typical penetration depth. In each frame, the check measures how far the deepest hand vertex is inside the object. A trajectory is rejected if this depth is above 20 mm in more than 10% of its frames, which removes gross failures such as a hand passing through the object. The retained trajectories are far below this limit. In the median trajectory, nine in ten frames stay below 5.0 mm, and in 74% of trajectories they stay below 10 mm. The final pose refinement (Appendix B.1) also shifts the hands by up to 8 mm to reduce remaining penetration. We do not push penetration to zero, because a strict limit is not free. Recorded grasps in the annotated motion capture overlap the object mesh by a few millimetres themselves (median 1.6 mm), and when we enforced zero penetration as a hard constraint on the hand pose, the fingertips were pulled 20–95 mm away from the observed grip and curled into a different grasp. Near contact, our reconstructed references penetrate less than the annotated datasets (0.08 vs 0.23 mm per hand vertex). With a 12 mm threshold, 81% of the trajectories would still pass, and we will release the per-trajectory statistics so users can filter more strictly.

Low quality of the generated motions. We agree that the motions our model generates have artifacts, and we measured how large they are. Over the 20 samples our model generates for every benchmark window and task, on frames where a predicted hand is within 2 cm of the predicted object, the hand sinks 0.37 mm into the object per vertex on average, and the predicted object moves with no fingertip within 2 cm in 25% of frames. The recorded reference trajectories of the same windows show some of this too, motion capture included (0.15 mm and 11%). Most forecasts still work physically, and 73.8% of them could be executed by a simulated robot hand in the SPIDER study above. They are also the closest to the recorded human motion among released forecasters (Table 2), and no other forecaster generates hands and objects together, so theirs cannot even be checked for contact. Additionally, training with our reconstructed data yields better performance than training with motion capture data alone (Table 1).

Best of 20 versus single predictions. Best of N is the standard metric for forecasters that draw samples [6, 7, 8], and Table 2 used it for every method. USST, EgoH4 and MotionForesight are deterministic, so their best of N is their single prediction. Because best of N can still favor methods that draw samples, we also scored every method on a single prediction, and methods that draw several samples with the energy score [9], which averages the error of all samples and subtracts half their mutual distance. It equals the plain error for a single prediction and is minimised in expectation only when the samples follow the distribution of real outcomes.

FDE (mm)Wrists L/R, single predictionWrists L/R, energy scoreObject, single predictionObject, energy score
USST (deterministic)196.3/182.3196.3/182.3––
ForeHand4D206.9/217.3122.8/133.6––
MotionForesight (deterministic)––142.9142.9
ObjectForesight––193.2130.4
Ours170.3/156.1103.4/92.4150.991.1

Ours has the lowest error in every column except one. For objects, the single prediction with the lowest final error is MotionForesight’s, but it barely moves the object (median 18 mm against 81 mm in the reference), and its error is within 1 mm of not moving at all. On the annotated windows our single prediction has the lower final object error (167.8 vs 185.8 mm).

References

[6] A. Gupta et al. Social GAN: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018.

[7] T. Salzmann et al. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.

[8] Y. Yuan and K. Kitani. DLow: Diversifying latent flows for diverse human motion prediction. In ECCV, 2020.

[9] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 102(477):359–378, 2007.

Reviewer_CEPrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a unified model for joint 3D hand–object motion prediction, including wrist motion, finger articulation, and rigid-object poses. It formulates forecasting, cross-entity prediction, goal-conditioned generation, and temporal infilling as masked completion tasks using flow matching. Hand–object relation tokens and contact constraints encourage coordinated motion. A reconstruction pipeline extracts 62,562 trajectories from internet videos for training alongside annotated datasets. Experiments evaluate prediction accuracy, component contributions, and task sharing, with additional robot-retargeting experiments assessing physical feasibility in simulation.

Critical strengths
  1. A common observation-mask formulation supports forecasting, cross-entity prediction, goal conditioning, and infilling. Comparisons with task specialists substantiate the value of sharing one model across these tasks.
  2. The extracted internet data improve forecasting. Adding them reduces single-sample object&hand FDE, supporting the practical value of the reconstruction pipeline.
Critical weaknesses
  1. The benefit of joint hand–object forecasting over separate prediction is not directly evaluated. The component ablations retain joint prediction, while the specialist comparisons assess task sharing. A comparison with separate hand and object predictors using the same training data and observations would clarify the gains in trajectory accuracy and contact consistency.
  2. The evaluation does not separate gains on independently annotated trajectories from gains on reconstructed references. Internet-video references receive 50% of the aggregate weight and are produced by the same pipeline used for training labels. It remains unclear how much of the reported data benefit transfers to independently annotated motion.
Questions for authors

With identical training data, observed context, and sample budget, does joint prediction improve trajectory accuracy and contact consistency over predicting hands and objects separately?

1 comment

Thank you for your positive assessment and for raising two specific questions. We conducted additional experiments to address both.

Joint versus separate prediction. Due to time constraints, we were unable to retrain separate hand-only and object-only models. Instead, since a hand-only or object-only model never sees the other entity, we removed the other entity's observations from our model. Its accuracy drops, but it stays ahead of the best released forecasters under the energy score [1], which scores the 20 samples together by averaging their errors and subtracting half their average distance to each other (mm, lower is better):

Object ADE/FDELeft/right wrist ADE
Ours55.5/91.167.5/65.0
Ours, other entity not observed59.9/102.671.3/72.5
ObjectForesight77.0/130.4–
ForeHand4D–92.4/103.1

Annotated versus reconstructed references. We now report on the two subsets separately. On the 256 annotated windows, which are independent of our pipeline, training with the internet trajectories improves the 20 samples in every task column. The energy score is 4–16% lower (lower is better) and the best-of-20 error 2–17% lower in all columns. The single prediction stays within 7 mm of the motion-capture-only model in every column (0.1 mm better on average). On internet videos, the single prediction also improves substantially, for example forecast hands 143.1 vs 205.2 mm and object→hands 89.9 vs 156.9 mm.

References

[1] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 102(477):359–378, 2007.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.