acceptodds
Under review as a conference paper at ICLR 2027

Generative Hand Modeling: A Unified Model for Egocentric Hand Motion Reconstruction, Infilling and Generation

Abstract

Modeling 4D hand motion is important for understanding human manipulation and learning from human demonstrations. Existing approaches, however, typically treat reconstruction, motion infilling, and generation as separate problems, despite their shared need to model hand articulation and temporal dynamics. We present Generative Hand Modeling (GHM), a unified framework that formulates these tasks as conditional hand motion modeling under varying levels of visual observability. GHM uses a shared motion representation and backbone to learn from different forms of visual and semantic supervision. It jointly models both hands within a common motion space. Experiments across these settings demonstrate strong reconstruction and temporal infilling performance, while retaining the ability to synthesize hand motion from scene and language conditions. Our results show the potential of unified generative modeling for learning hand motion from diverse sources of egocentric interaction data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand Motion Generation, 4D Reconstruction, Egocentric

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cd3POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GHM, a unified model for hand motion reconstruction, infilling, and text-conditioned generation.

The model is trained on large-scale hand motion and egocentric datasets. Experimental results demonstrate significant improvements in hand motion reconstruction, generation, and infilling tasks.

Critical strengths
  1. The authors have a clear primary objective: to achieve hand motion reconstruction, infilling, and generation within a unified framework. The overall design is centered around this objective.
  2. The paper presents extensive experimental comparisons with existing models across multiple tasks and datasets. The overall experimental evaluation is comprehensive. The supplementary material further demonstrates the effectiveness of GHM through qualitative results.
  3. The experimental results demonstrate the strong performance of GHM. The unified framework outperforms existing models on reconstruction, infilling, and generation tasks, validating the effectiveness of the proposed approach.
Critical weaknesses
  1. Limited novelty. The main contribution of this paper is to integrate Hand Motion Reconstruction, Infilling, and Generation into a unified framework. However, most of the model's core technical components are based on existing methods, with limited novel methodological contributions.
  2. Insufficient ablation studies. This paper uses large-scale training data to jointly train three tasks within a unified framework. However, the authors do not provide sufficient ablation studies to demonstrate to what extent the performance improvements come from large-scale training and to what extent they benefit from joint supervision across the three tasks.
  3. Limited analysis of model complexity and efficiency. This paper proposes a generative model based on a DiT architecture and diffusion, which generally implies a larger model and higher inference overhead. The authors report the number of Transformer layers, but could they provide more detailed information about the model's parameter count and inference speed? If possible, I would also appreciate comparisons with other existing methods. This would help potential users assess the additional computational cost associated with the performance improvements.
Questions for authors

Could the authors provide further analysis to clarify to what extent the performance improvements of GHM come from large-scale training data, and to what extent they benefit from joint training across Reconstruction, Infilling, and Generation?

Reviewer_cxCoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to treat egocentric 4D hand-motion reconstruction, temporal infilling, and scene/language-conditioned generation as one conditional modeling problem under varying visual observability.

GHM adapts GENMO's dual-mode (diffusion + single-pass regression) Transformer to a dual-hand MANO representation expressed in both camera and first-frame world coordinates, links the two with a camera--world consistency loss, and uses condition availability masks so that eight heterogeneous egocentric datasets (~43.4M frames) can supervise one model.

It reports the lowest camera-space MPJPE on HOT3D (18.19 vs. 24.51 mm for HandFlow) and competitive results on unseen H2O, better matched-gap infilling than HaWoR and Dyn-HaMR on HOT3D and ARCTIC, lower errors than LatentAct for generation on HoloAssist, and ablations of dual-mode training and the consistency loss.

Critical strengths
  1. Strong reconstruction and infilling results under a carefully matched protocol. All camera-space methods are scored on the same valid frames, and all except the detector-free MINT share the same HaWoR-YOLO detections (Sec. 4.1), and GHM lowers HOT3D MPJPE from 24.51 to 18.19 mm (25.8% relative to HandFlow) while remaining competitive on the unseen H2O dataset (Table 1). For infilling, GHM is compared on exactly the gaps each baseline can handle (GHM_H, GHM_D) and still lowers all four error metrics, while covering 99.1% of HOT3D and 100% of ARCTIC gaps (Table 5, Appx. C.3). This careful protocol is consequential because it makes the reconstruction-side gains credible.

  2. A practical recipe for pooling heterogeneous egocentric data into one hand-motion model. Per-condition availability masks plus random condition dropout let datasets with very different annotations (video, MANO, text, camera) supervise a single model without task tokens (Sec. 3.2), and dual-mode training improves reconstruction over regression alone (Table 6). Combined with explicit dual-hand modeling without flipping, this is a useful and reusable design for learning from large-scale egocentric demonstrations.

Critical weaknesses
  1. The generation side, which is central to the "unified generative" claim, is weakly evaluated. On HoloAssist, generation is compared against a single learned baseline, LatentAct, run in 30-frame windows that are concatenated "without contact registration or cross-fading" (Appx. C.1), whereas GHM generates the full action in one rollout and also receives the ground-truth initial MANO states of both hands (Sec. 3) and, in one variant, the complete ground-truth 2D wrist trajectory. No metric measures adherence to the language instruction, even though the task is described as language-conditioned generation, and GHM's root-relative pose diversity is lower than LatentAct's (Table 3). In addition, 52 of the 60 candidate HoloAssist evaluation actions come from training videos (Appx. B.3; 59 are used in Table 3), and the TACO comparison (Table 4) is against a fixed-pose wrist-following heuristic. Text-to-hand-motion methods that the paper discusses, such as CLUTCH (Thambiraja et al., 2026) and the HandX benchmark (Zhang et al., 2026), are not compared. This matters because, without credible generation results, the paper mainly demonstrates a strong reconstruction/infilling model.

    • Suggestion: compare with LatentAct under matched conditioning and sequence length, add a metric for instruction adherence, evaluate on actions not seen in training, and compare with a text-to-hand-motion method such as CLUTCH.
  2. The contribution beyond GENMO and UniHand is unclear, and the native world-space branch underperforms. The architecture and dual-mode training follow GENMO (Li et al., 2025a), and UniHand (Sun et al., 2026) already unifies estimation and generation for hands; beyond the dual-hand camera/world representation, the consistency loss, a scene cross-attention pathway, and condition masking, the paper does not position GHM's methodological advance against these two works, and compares with UniHand only through its published world-space numbers (Table 2). In that table, GHM's directly decoded world-space trajectory (C0 rollout) is worse than HaWoR on WA-MPJPE (19.23 vs. 16.24 mm); the best result comes from transforming camera-space predictions with ViPE extrinsics (Cam2World), the same post-hoc transform that the paper applies to HaWoR's camera-space predictions in that table. The ablations are also run in unspecified configurations: GHM's HOT3D MPJPE is 18.19 mm in Table 1 but 20.58 and 19.27 mm for the full settings in Tables 6 and 7, and no variance is reported. This matters because the unified world-space representation is presented as a core design choice.

    • Suggestion: state the methodological difference from GENMO and UniHand explicitly, report the C0 and Cam2World results separately in the main claims, and specify the configuration of each ablation with variance.
Questions for authors

Clarifications (could change my assessment):

  1. What are the methodological differences from GENMO and UniHand beyond the dual-hand camera/world representation, the consistency loss, scene cross-attention, and condition masking? Were the UniHand numbers in Table 2 obtained on the same HOT3D clips and valid frames?
  2. How does GHM compare with LatentAct when both receive the same conditions (e.g., GHM without the GT initial MANO states), and how does it score on the 8 HoloAssist actions from validation videos alone?
  3. How is adherence to the text instruction evaluated? Do you have any text--motion alignment metric (e.g., retrieval precision) or a user study?
  4. Why do the full-model numbers differ between Table 1 and the ablations in Tables 6--7 (18.19 vs. 20.58/19.27 mm)? Which checkpoints and settings do the ablations use?

I am open to revisiting my rating if the rebuttal addresses these points well.

Reviewer_vDNdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a hand-motion modeling framework GHM, which jointly supports reconstruction, motion infilling, and generation within a unified motion space. GHM leverages this shared motion representation and backbone to learn from different visual and semantic inputs, performing hand reconstruction from scene, hand observations, camera motion, hand motion generation from scene and motion description, and motion infilling from partial hand observations. As for training objectives, it further incorporates a cross-representation consistency loss to ensure world- and camera-space hand predictions are consistent except for the generative and reconstruction losses. Extensive experiments demonstrate the effectiveness of GHM on egocentric hand-object interaction datasets.

Critical strengths

The presentations on introduction and related works are clear and convincing. The motivation of this paper is to build a common motion space that different tasks could share and complement each other.

The main framework is overall clear, unifying three hand motion tasks by varying different conditioning information. To ensure world- and camera-space alignment, an additional cross-representation consistency loss is introduced, whose effectiveness is validated in experiments.

Critical weaknesses

Some previous methods like UniHand also introduce similar ideas to use a unified hand motion space for 4D hand motion reconstruction and generation. Since UniHand is not opensource, it is understandable that GHM does not include a direct comparison. A key difference between GHM and UniHand is that GHM introduces text conditioning as a semantic input. Beyond enabling language-guided interactions, the ablation studies do not investigate whether the additional text conditioning improves hand motion modeling, which I think is worth investigating to see whether semantic information could improve motion modeling as well.

The presentation of the method and experiments could be improved. There is no reference for Figure 3-6 in the main paper. Discussion on the qualitative results are encouraged. Some representations are not consistent, like “Tab.” used in the main paper while “Table” used in Appendix. “Eq. equation” double equation representation in L220. It should be “.” instead of “,” in Equation (2). “RoPE-based Transformer denoiser with bidirectional self-attention” is an unnatural representation since RoPE is a commonly used positional encoding scheme for Transformer, and bidirectional self-attention is a natural choice for tasks that do not require causal attention. This emphasis is a bit unusual.

Questions for authors

There are extensive experimental results provided in the paper. However, I think there is little discussion regarding the experimental results. Improvements are significant, but what implications and observations do authors discover that could benefit future research? Currently, I lean to weak accept and I will revisit my rating during the rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

Dear SACs, ACs, and Reviewers,

We sincerely thank all reviewers for their insightful feedback and constructive suggestions.

We appreciate their recognition of the clear motivation and unified formulation of hand-motion reconstruction, infilling, and generation (Reviewers vDNd and cd3P), as well as the comprehensive experimental evaluation and strong performance across multiple tasks and datasets (Reviewers cxCo and cd3P). We also appreciate the recognition of the carefully matched evaluation protocols and the practical value of learning from heterogeneous egocentric datasets (Reviewer cxCo), and the effectiveness of the cross-representation consistency objective (Reviewer vDNd).

In response to the reviewers' constructive feedback, we have further clarified the methodology and strengthened the experimental analysis. The improvements are summarized below:

  • Further clarification of GHM's methodological distinctions from existing approaches, including native bimanual modeling, scene-grounded interaction generation, and camera–world consistent motion representation (Reviewers cxCo and cd3P).
  • Further analysis of the role of text conditioning under incomplete visual observations, together with an expanded discussion of experimental findings and their implications (Reviewer vDNd).
  • Additional ablation studies on training data and supervision of GHM (Reviewers cxCo and cd3P).
  • Extended generation evaluation, including a matched-condition comparison with LatentAct, VLM-based evaluation and a user study comparing GHM with CLUTCH, and additional comparisons with HandX (Reviewer cxCo).
  • Additional evaluation of training- and inference-seed variability, together with measurements of model complexity and inference efficiency (Reviewers cxCo and cd3P).

We have incorporated these clarifications and additional analyses into the revised manuscript, with expanded discussions of experimental findings and improved presentation.

Once again, we sincerely thank the SACs, ACs, and reviewers for their time and valuable feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.