acceptodds
Under review as a conference paper at ICLR 2027

G4D-Hand: Geometry-grounded 4D hand trajectory estimation from egocentric videos

Abstract

Egocentric 4D hand trajectory estimation aims to recover the 3D positions of hand keypoints over time in a consistent coordinate frame from egocentric video. Most existing methods follow an articulation-centric pipeline in which hands are detected and cropped, reconstructed from independent image patches, assigned wrist translation through a weak-perspective camera, and subsequently placed in the world by an external SLAM system. At no stage do these pipelines represent the hands in relation to the scene in which they move, even though this could help resolve hand depth and occlusion. We argue that scene geometry should shape the hand representation from the outset and introduce G4D-Hand, which grounds hand prediction in a geometric representation. It reconstructs both hands from shared full-frame features, without a hand detector or hand-specific image encoder, jointly recovering articulation and metric wrist translation in a common camera frame. Inter-hand and temporal supervision further encourage consistent relative positioning and coherent motion, and the same backbone estimates camera motion to lift predictions into the world frame without a separate SLAM stage. G4D-Hand runs at 30+FPS for camera-frame, matches state-of-the-art articulation accuracy, and substantially improves localization on egocentric benchmarks. On ARCTIC, a dataset with frequent hand occlusions, it reduces inter-hand relative wrist error by 50% and camera-frame hand trajectory error by 31%, against the strongest baseline on each metric. Code and checkpoints will be made public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric vision, hand pose estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_coiuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes G4D-Hand, a method for 4D hand trajectory estimation from egocentric video. It is built on a frozen 4D reconstruction backbone, CUT3R. Instead of detecting and cropping hands, left- and right-hand branches extract hand features directly from full-frame CUT3R features, which are conditioned on camera intrinsics through a raymap. A temporal transformer refines these features. A HaMeR-initialized decoder then predicts MANO pose and shape, and a separate head predicts metric wrist translation from CUT3R’s feature pyramid. A second CUT3R pass estimates the camera motion, which lifts the hands to the world frame without external SLAM. The model is trained on HOT3D, ARCTIC, and H2O and evaluated on the same three datasets. It achieves the lowest trajectory, wrist, and inter-hand errors, with articulation accuracy comparable to the state of the art.

Critical strengths
  1. Clear idea and simple design. The paper tests a clear hypothesis: hand reconstruction should be grounded in scene geometry. It estimates both the hands and the camera with a single frozen CUT3R backbone, without a hand detector, a hand-specific image encoder, or external SLAM.
  2. Large gains in localization and trajectory accuracy. On all three benchmarks, the method achieves the lowest trajectory error (G-MPJPE), wrist error, and inter-hand error (MRRPE). On ARCTIC, for example, it reduces G-MPJPE by 31% and MRRPE by about 50% relative to the strongest baseline for each metric. Articulation accuracy (RA-MPJPE) remains comparable to the best existing methods.
  3. Robustness to occlusion. Appendix D.3 shows that the method has the lowest wrist error and MRRPE in every occlusion bin on ARCTIC, including when 75–100% of the joints are occluded.
  4. Efficiency. The model trains only 27M parameters, and camera-frame prediction runs at over 30 FPS on a single RTX 4090.
Critical weaknesses
  1. Possible advantage from in-domain training. G4D-Hand is trained only on the three evaluation datasets (HOT3D, ARCTIC, and H2O). The baselines are evaluated with their public checkpoints and are not retrained on the same data. Part of the gain may therefore come from in-domain training. There is also no evaluation on unseen datasets.
  2. No long-sequence evaluation with estimated cameras. The CUT3R state is reset every 16 frames, or every 64 frames for camera estimation. World-frame results with estimated camera motion cover only 64-frame clips. Full-sequence results (Table 8) are reported only with ground-truth camera poses, so the full pipeline is not verified on long videos.
  3. Unclear handling of invisible hands. The method uses no hand detector, but the paper does not explain what it outputs when a hand is out of view or only one hand is visible, or whether it predicts hand presence. The evaluation also covers only in-view hands (Appendix C.1).
  4. Limited novelty.
  • Because the reference pose is fixed, the raymap only encodes camera intrinsics. This serves a similar purpose to the intrinsics inputs in CLIFF and WildHands, and the paper does not compare against these alternatives.
  • The method relies on the pretrained HaMeR MANO decoder, and the paper does not analyze how much this hand-specific prior contributes.
  • Conceptually, it is close to Human3R, which also builds on CUT3R. The main difference is that G4D-Hand removes the hand-specific feature encoder (Human3R uses Multi-HMR features).
Questions for authors
  1. G4D-Hand is trained only on HOT3D, ARCTIC, and H2O, while the baselines are evaluated with their public checkpoints. How much of the improvement comes from this in-domain training? Can the authors report (a) results of strong baselines (e.g., WiLoR) fine-tuned on the same data, or (b) results of G4D-Hand on a dataset not used for training (e.g., OakInk2 or HOI4D)? (could change my assessment)
  2. World-frame results with estimated cameras are limited to 64-frame clips. How does the full pipeline perform on longer sequences? How are 64-frame chunks, each with its own CUT3R state and scale estimate, connected into a full trajectory? (could change my assessment)
  3. What does the model output when a hand is out of view or only one hand is visible? Does it predict hand presence? How are such frames handled in training? Evaluation covers only in-view hands (Appendix C.1). (clarification)
  4. The raymap uses a fixed reference pose, so it encodes only the camera intrinsics. Have the authors compared it with simpler ways of providing intrinsics, such as CLIFF- or WildHands-style inputs? (clarification)
  5. How much does performance depend on the pretrained HaMeR MANO decoder (e.g., compared with a randomly initialized decoder)? Beyond removing the hand-specific encoder, what is the main technical difference from Human3R? (clarification)
4 comments
AuthorsOpenReview ↗

Generalization to unseen datasets, extended world-frame evaluation, and robustness to hand prior initialization

We thank the reviewer for their valuable feedback and positive evaluation of our work. We appreciate their support for grounding hand reconstruction in scene geometry. We also thank the reviewer for highlighting our model’s efficiency, robustness to occlusion, and strong localization and trajectory accuracy as key strengths.

W1., Q1. Generalization to unseen datasets

We evaluate on two datasets unseen in training: SHOW3D (participant XYZ109 validation sequences, ~26k frames) and HoloAssist (152 randomly selected 128-frame clips). We compare G4D-Hand with ReViV (ECCV26) and EgoForce (SIGGRAPH26), both trained on at least HOT3D, H2O, and ARCTIC. These results support that G4D-Hand generalizes beyond the training datasets.

DatasetModelG-MPJPE ↓GA-MPJPE ↓Wrist ↓MRRPE ↓RA-MPJPE ↓Acc. ↓
SHOW3DEgoForce160.2104.9143.2200.963.444.5
ReViV157.981.8138.9177.759.68.3
G4D-Hand62.432.547.147.331.15.2
HoloAssistEgoForce109.653.571.089.336.913.4
ReViV67.335.843.949.726.36.1
G4D-Hand47.526.566.547.732.83.6

Table 1: Camera-frame evaluation. Best zero-shot results bolded. ReViV is trained on HoloAssist and is excluded from its zero-shot ranking.

G4D-Hand achieves the lowest trajectory errors on both datasets, reducing G-/GA-MPJPE by 29.4%/26.0% on HoloAssist and ~60% for both on SHOW3D.

W2., Q2. Longer-window world-frame evaluation

We extend the world-frame evaluation to 128-, 192- and 240-frame windows. For this experiment, CUT3R processes 128-frame chunks with a 16-frame overlap. Each chunk of camera poses is independently scaled to absolute metric with hand-to-scene depth ratio (Sec. 4.2), then rigidly aligned to its predecessor using shared camera poses, with overlapping poses blended. This recovers metric camera trajectories in the first chunk's reference frame without ground truth.

MethodG-128 ↓GA-128 ↓G-192 ↓GA-192 ↓G-240 ↓GA-240 ↓
G4D-Hand80.144.293.452.093.355.3
G4D-Hand + SLAM64.136.471.842.574.745.4
HaWoR94.249.895.752.898.955.8

Table 2: World-frame evaluation on ARCTIC with estimated camera poses. G-/GA- stand for G-MPJPE and GA-MPJPE and suffixes denote window lengths. Best results bolded.

G4D-Hand with SLAM achieves the lowest errors across all temporal windows. Performance of G4D-Hand with (native) CUT3R camera poses is better or comparable with HaWoR, which uses adapted DROID-SLAM.

W3., Q3. Clarification on out-of-view hands

The model does not predict hand presence. Training uses 16-frame windows with both hands present (five or more 2D keypoints per hand within the frame). All models are evaluated on in-view hands, whether one or both are visible, consistent with H2O/ARCTIC protocols. We plan to explore scene-grounded hand presence prediction, out-of-view and more than two hand pose prediction in the future work.

W4., Q5. Influence of pretrained hand priors in MANO regression head

We analyze the influence of initialization of MANO regression head with weights from HaMeR MANO decoder with two controlled ablations. The results below suggest that this initialization is not the main driver of performance.

In the default G4D-Hand setting, the MANO regression head is initialized from HaMeR, with only selected parameters, mainly cross-attention layers, reinitialized and trained. Here, we compare it with two new settings: FT (fine-tuning the full head), and Reinit. from Scratch (training the head from random initialization). All variants use the same architecture and training data. We shorten training from 100 to 30 epochs due to rebuttal time and resource constraints.

SettingEpochsG-MPJPE ↓GA-MPJPE ↓Wrist ↓MRRPE ↓RA-MPJPE ↓Acc. ↓
Default3057.724.139.745.420.92.7
FT3054.923.740.446.320.72.7
Reinit. from Scratch3054.223.339.946.520.42.7

Table 3: MANO regression head ablation evaluated in the camera frame on the HOT3D.

G4D-Hand with both ablation settings performs similarly to the default, with slightly higher MRRPE and wrist error and lower trajectory errors.

W4., Q4., Q5. Comparisons with related works

W4, Q4: Intrinsics conditioning. G4D-Hand conditions full-frame rather than crop features using CUT3R's pretrained raymap encoder to predict metric wrist translation. Both G4D-Hand and WildHands train on ARCTIC. On its validation set, ours achieves lower errors across all metrics, reducing wrist error by ~2x. We will add these results and discuss both works in the revision.

W4., Q5: Human3R. G4D-Hand keeps CUT3R frozen and uses raymap conditioning and a dedicated wrist regression head to predict metric wrist positions from its geometric features. Hand branches localize hands within full-frame features through learned queries, without a detector.

Thank you for the thorough response. The new results on SHOW3D and HoloAssist address my concern about in-domain training, and the MANO-head ablation shows that the HaMeR initialization is not the main source of the gains. The clarification on out-of-view hands is helpful; please state this limitation explicitly in the paper. Two points remain partly open: for longer windows, the native pipeline performs on par with HaWoR at 192–240 frames, and the best results rely on external SLAM, which should be clarified given the paper’s no-SLAM claim; and the comparison with WildHands should include the numbers and, ideally, a controlled ablation replacing the raymap with CLIFF-style inputs. Overall, my concerns are largely resolved, and I keep my positive assessment.

Thank you for the thorough response. The new results on SHOW3D and HoloAssist address my concern about in-domain training, and the MANO-head ablation shows that the HaMeR initialization is not the main source of the gains. The clarification on out-of-view hands is helpful. Please state this limitation explicitly in the paper. Two points remain partly open: for longer windows, the native pipeline performs on par with HaWoR at 192–240 frames, and the best results rely on external SLAM, which should be clarified given the paper’s no-SLAM claim. In addition, the comparison with WildHands should include the numbers and, ideally, a controlled ablation replacing the raymap with CLIFF-style inputs. Overall, my concerns are largely resolved, and I keep my positive assessment.

Thank you for your response and positive evaluation of our work! We are happy that our rebuttal largely resolved your concerns, including the evaluation on unseen datasets and the MANO head initialization ablation.

We have uploaded a revised manuscript. We expanded the related work (Section 2) with a more detailed comparison to WildHands, CLIFF, and Human3R. We added quantitative evaluations on datasets not used during training in Appendix D.1 (Table 9) and qualitative examples in Appendix E.2 (Figure 11). We also added the world frame evaluation over longer temporal windows in Section 4.2 (Table 3). We included the MANO head initialization ablation in Section 4.4 (Table 6). In the limitations (Section 5), we added hand presence prediction to our future work, together with handling an arbitrary number of hands and out-of-view hands. We also included WildHands results on ARCTIC in Table 1 and Table 11.

Reviewer_7w13OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors proposed G4D-Hand, a method to estimate 4D hand trajectories from monocular videos. The proposed method rely on a pre-trained 4D scene reconstruction backbone to estimate a hand pose, a wrist position and a camera motion. Motion consistency and hand positions are refined via camera raymap and temporal refinement steps. Experiments are conducted using ARCTIC, H2O and HOT3D and it demonstrates the improvement in hand localization and trajectory estimation.

Critical strengths
  • Strong performance on multiple benchmarks: The method achieves improvements in hand localization and trajectory estimation accuracies on ARCTIC, H2O, and HOT3D.

  • New lessons from geometric representation usage: The framework estimates hand articulation and wrist translation directly from geometric features, eliminating the need for separate hand detectors and SLAM. This provides an interesting lesson by rolling as a practical alternative to conventional pipelines.

  • Real-time efficiency: The method achieves over 30 FPS.

Critical weaknesses
  • Insufficient ablations for effectiveness of geometric feature: The paper argues that pre-trained scene geometry representations provide an effective foundation for hand trajectory estimation. However, it seems not easy to say whether the performance improvements come from the geometric features or from other design choices such as raymap conditioning or temporal processing.

  • Limited analysis on pretrained hand priors. The MANO regression head is initialized from HaMeR, with most of its parameters frozen. More analysis could be conducted on the effect of the initialization. For example, how it is affected by different pose estimators or how it is affected by unfreezing parameters.

  • Limited evaluation for long-term consistency. The world-frame evaluation is conducted on relatively short sequences of 64 frames. Additional evaluations on longer sequences would strengthen the claims regarding consistent world-space hand motion estimation.

Questions for authors
  • In Table 2, I am curious about the results that use SLAM, in addition to the results obtained by adding CUT3R to the compared models (HaMeR and WiLoR).

  • Figure 4 evaluates the long-term consistency of hand estimation using ground-truth camera extrinsic parameters. Since the proposed method uses scene geometry features and also estimates camera motion with the same backbone, it might be necessary to examine whether this stability is maintained when using camera poses estimated by CUT3R. How's the opinion on this?

4 comments
AuthorsOpenReview ↗

Benefits of geometric features, robustness to hand prior initialization, and extended world-frame evaluation

We thank the reviewer for the valuable feedback and for recognizing metric hand pose prediction from scene geometry features as an interesting, practical alternative to typical hand-centric pipelines. We also thank the reviewer for highlighting strong performance and efficiency of G4D-Hand.

W1. Benefits of geometric features

We design controlled ablation experiments to compare the geometric backbone (CUT3R) with an equally sized visual backbone (ImageNet ViT-L), with and without raymap conditioning. Under matched training, CUT3R backbone improves wrist and trajectory accuracy in both settings, suggesting the benefits of geometric backbone features.

All four variants use the same architecture (apart from backbone) and training protocol. The raymap encoder uses CUT3R weights and its encoding is added to the backbone features. We shorten training from 100 to 30 epochs due to rebuttal resource and time constraints.

On the HOT3D dataset, CUT3R reduces wrist error and GA-MPJPE by ~14% and ~12% without raymaps, and ~8% and ~7% with raymaps, respectively. Even with the visual backbone and only 30 training epochs, G4D-Hand substantially outperforms ReViV, which also trains on HOT3D.

MethodBackboneEpochsRaymapG-MPJPE ↓GA-MPJPE ↓Wrist ↓MRRPE ↓RA-MPJPE ↓Acc. ↓
G4D-HandCUT3R30✓57.724.139.745.420.92.7
G4D-HandViT-L30✓60.426.043.044.422.23.1
G4D-HandCUT3R30✗67.129.148.447.222.62.7
G4D-HandViT-L30✗72.333.056.253.223.73.3
G4D-HandCUT3R100✓48.621.036.742.318.72.5
ReViV---121.451.885.192.939.25.1

Table 1: Backbone ablation on HOT3D. The 100-epoch model and ReViV are results from the manuscript.

These experiments complement the ablations of temporal modeling and raymap conditioning (manuscript Table 4) and the inter-hand loss (manuscript Table 9).

W2. Influence of pretrained hand priors in MANO regression head

We analyze the influence of initialization of MANO regression head with weights from HaMeR MANO decoder with two controlled ablations. The results below suggest that this initialization is not the main driver of performance.

In the default G4D-Hand setting, the MANO regression head is initialized from HaMeR, with only selected parameters, mainly cross-attention layers, reinitialized and trained. Here, we compare it with two new settings: FT (fine-tuning the full head), and Reinit. from Scratch (training the head from random initialization). All variants use the same architecture, training data, and 30-epoch protocol.

SettingEpochsG-MPJPE ↓GA-MPJPE ↓Wrist ↓MRRPE ↓RA-MPJPE ↓Acc. ↓
Default3057.724.139.745.420.92.7
FT3054.923.740.446.320.72.7
Reinit. from Scratch3054.223.339.946.520.42.7

Table 2: MANO regression head ablation evaluated in the camera frame on the HOT3D.

G4D-Hand with both ablation settings performs similarly to the default, with slightly higher MRRPE and wrist error and lower trajectory errors.

W3., Q1., Q2. Extended world-frame evaluation

Q1. Evaluation with SLAM. We add results with metric camera trajectories estimated with DROID-SLAM and Metric3D. Using SLAM poses yields lower errors than when using CUT3R poses. Still, even with its native CUT3R poses, G4D-Hand outperforms the baselines with SLAM on all metrics.

MethodG-MPJPE-sim ↓GA-MPJPE-sim ↓G-MPJPE ↓GA-MPJPE ↓
HaMeR + SLAM87.832.998.660.4
WiLoR + SLAM78.631.684.054.0
G4D-Hand + SLAM52.324.153.529.7
G4D-Hand58.129.558.432.1

Table 3: World-frame evaluation on the ARCTIC validation set (64-frame windows). -sim denotes similarity alignment, other metrics use rigid alignment. Best results are bolded.

W3., Q2. Evaluation over longer temporal windows We extend the world-frame evaluation to 128-, 192- and 240-frame windows. For this experiment, CUT3R processes 128-frame chunks with a 16-frame overlap. Each chunk of camera poses is independently scaled to absolute metric with hand-to-scene depth ratio (Sec. 4.2), then rigidly aligned to its predecessor using shared camera poses, with overlapping poses blended. This recovers metric camera trajectories in the first chunk's reference frame without ground truth.

MethodG-128 ↓GA-128 ↓G-192 ↓GA-192 ↓G-240 ↓GA-240 ↓
HaWoR94.249.895.752.898.955.8
G4D-Hand + SLAM64.136.471.842.574.745.4
G4D-Hand80.144.293.452.093.355.3

Table 4: World-frame evaluation on ARCTIC with estimated camera poses. G-/GA- stand for rigid G-MPJPE and GA-MPJPE and suffixes denote window lengths. Best results bolded.

G4D-Hand with SLAM achieves the lowest errors across all temporal windows. Performance of G4D-Hand with (native) CUT3R camera poses is better or comparable with HaWoR, which uses adapted DROID-SLAM.

We thank the reviewer for both of these suggestions as these results further improve our method.

Thank you again for your valuable review! We sincerely appreciate the time you spent evaluating our work and helping us improve it. As the discussion period is coming to an end, we would be grateful if you could let us know whether our responses have addressed your concerns. We would be happy to discuss any remaining questions.

We have uploaded a revised manuscript that includes the experiments from our rebuttal responses. We included the backbone and MANO-head initialization ablations in Section 4.4 (Table 6). We also extended the world-frame evaluation to longer temporal windows in Section 4.2 (Table 3) and added comparisons with baselines using SLAM-estimated camera poses in Appendix D.5 (Table 12).

Thanks for the author response. Authors effectively answered for my concerns. I am happy to raise my rating to weak accept.

Thank you for your positive response and evaluation of our work! We are delighted that our rebuttal has successfully addressed your concerns.

Reviewer_94jCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents G4D-Hand, a method for recovering 4D hand trajectories from monocular egocentric videos using a frozen scene reconstruction backbone. It extracts representations for both hands from full-frame CUT3R features, adapts a pretrained MANO decoder for articulation, and predicts metric wrist translation from geometric features conditioned on camera intrinsics. Temporal refinement and inter-hand supervision encourage coherent predictions. A separate decoder pass estimates camera motion for reconstruction in world coordinates without external SLAM. Experiments on ARCTIC, H2O, and HOT3D demonstrate substantial improvements in localization and trajectory accuracy while maintaining competitive articulation accuracy.

Critical strengths
  1. Well-Motivated Problem Formulation. I find the motivation of this work clear and compelling. Existing hand-centric approaches typically reconstruct hand articulation from cropped hand images, which can discard useful geometric context for estimating metric depth and relative hand positions. The authors address this limitation by leveraging pretrained 4D scene representations to jointly estimate hand articulation and localization. I think this shift from hand-centric to scene-grounded reconstruction is a sensible and promising direction.

  2. Effective Use of Pretrained Geometric Representations. I appreciate how the authors leverage pretrained geometric representations for hand reconstruction. By building on a shared CUT3R backbone, the proposed framework estimates hand poses, 3D positions, and camera trajectories without relying on separate hand detectors, dedicated hand image encoders, or external SLAM systems. In particular, directly regressing metric wrist translation from geometric features offers an appealing alternative to depth estimation based on weak-perspective camera models. The overall design is relatively straightforward, and the additional 27M trainable parameters seem reasonable.

Critical weaknesses
  1. Insufficient Evidence for the Benefits of Geometry-Grounded Representations. My main concern is that the experiments do not fully establish how much of the improvement comes from the geometric representations themselves. Compared with previous methods, G4D-Hand introduces several changes at once, including the use of CUT3R, full-frame features, raymap conditioning, a new translation regression head, and additional temporal and inter-hand supervision. While the overall results are impressive, it is difficult to tell which of these components contributes most to the gains. I would have liked to see a more controlled comparison with a conventional visual backbone using full-frame features, while keeping the remaining architecture and training setup unchanged. This would provide stronger evidence for the paper's central claim.

  2. Unclear Latency in Real-Time Settings. Although the reported throughput of 31–39 FPS is encouraging, I am not entirely convinced that it reflects the practical latency of the full system. The temporal module uses bidirectional attention over non-overlapping 16-frame windows, meaning that predictions for earlier frames depend on future observations. At 30 FPS, this alone could introduce up to approximately 500 ms of waiting time before computation. Moreover, the higher throughput of 39 FPS is achieved through batched processing, which does not necessarily translate to low latency in a streaming setting. I also noticed that the reported runtime focuses on camera-frame prediction, while world-frame reconstruction requires an additional CUT3R decoder pass. A more complete latency analysis would make the efficiency claims more convincing.

Questions for authors

Overall, I lean toward a weak acceptance given the clear motivation and strong empirical results, but I still have some reservations about the evidence supporting the main claims. Addressing the following questions would strengthen my confidence in the work.

  1. Contribution of Geometric Representations. Could the authors compare CUT3R with a conventional visual backbone that extracts full-frame features, while keeping the remaining architecture, training data, and losses unchanged? I believe this would help clarify how much of the performance gain can be attributed specifically to pretrained geometric representations.

  2. Inference Latency and Efficiency. Could the authors report the actual end-to-end latency in a streaming setting, including the additional computation required for world-frame reconstruction? I would also be interested in seeing how the accuracy and latency change with different temporal window sizes, particularly given the reliance on future frames.

3 comments
AuthorsOpenReview ↗

Benefits of geometric features and extended runtime analysis

We thank the reviewer for their valuable feedback and supportive review. We appreciate their positive assessment of our idea formulation and the recognition of our proposal to shift from hand-centric to scene-grounded hand reconstruction as a promising direction for the hand pose estimation research.

W1., Q1. Benefits of geometric features

We design controlled ablation experiments to compare the geometric backbone (CUT3R) with an equally sized visual backbone (ImageNet ViT-L), with and without raymap conditioning. Under matched training, CUT3R backbone improves wrist and trajectory accuracy in both settings, suggesting the benefits of geometric backbone features.

All four variants use the same architecture (apart from backbone) and training protocol. The raymap encoder uses CUT3R weights and its encoding is added to the backbone features. We shorten training from 100 to 30 epochs due to rebuttal resource and time constraints.

On the HOT3D dataset, CUT3R reduces wrist error and GA-MPJPE by ~14% and ~12% without raymaps, and ~8% and ~7% with raymaps, respectively. Even with the visual backbone and only 30 training epochs, G4D-Hand substantially outperforms the recent ReViV, which also trains on HOT3D.

MethodBackboneEpochsRaymapG-MPJPE ↓GA-MPJPE ↓Wrist ↓MRRPE ↓RA-MPJPE ↓Acc. ↓
G4D-HandCUT3R30✓57.724.139.745.420.92.7
G4D-HandViT-L30✓60.426.043.044.422.23.1
G4D-HandCUT3R30✗67.129.148.447.222.62.7
G4D-HandViT-L30✗72.333.056.253.223.73.3
G4D-HandCUT3R100✓48.621.036.742.318.72.5
ReViV---121.451.885.192.939.25.1

Table 1: Backbone pretraining ablation on HOT3D. The original 100-epoch model and ReViV results from the submitted manuscript are included for reference.

W2., Q2. Extended speed and streaming latency analysis

We extend the speed analysis to include end-to-end streaming latency and world-frame reconstruction. We evaluate G4D-Hand and six other methods on the same hardware (NVIDIA RTX 5090 GPU with 32 GB; Intel Core Ultra 7 265K CPU; 64 GB RAM) on 10 ARCTIC (CVPR23) clips of 128 frames each. Among temporal methods, G4D-Hand achieves the highest FPS and lowest latency.

G4D-Hand achieves 34.5 FPS with 0.71 s average latency for camera-frame estimation, 29.6 FPS with 0.79 s latency for world-frame estimation using up-to-scale camera poses, and 26.5 FPS with 1.14 s latency for metric world-frame estimation. World-frame timings include the additional CUT3R camera-pose decoder and, for metric reconstruction, the pointmap decoder. Metric camera-translation scale is estimated from the first 16-frame window using the robust procedure in Sec. 4.2. The additional cost of metric reconstruction could be reduced by predicting camera-frame pointmaps only for frames used in scale estimation. G4D-Hand processes windows sequentially, without batching.

MethodCameraWorldProcessingms/frame ↓FPS ↑Latency [s] ↓
HaMeR✓—Per-frame5032.00.50
WiLoR✓—Per-frame3132.60.03
EgoForce✓—Per-frame4323.20.04
HaPTIC✓—Whole video3223.143.40
HaWoR✓✓ (metric)Whole video1059.615.50
Dyn-HaMR✓✓Whole video21110.5272.30
G4D-Hand✓—16 frames2934.50.71
G4D-Hand✓✓16 frames3429.60.79
G4D-Hand✓✓ (metric)16 frames3826.51.14

Table 2: Speed and end-to-end latency on identical hardware at 30 FPS input. Camera/World indicate output coordinates. Processing indicates number of frames processed together. Latency includes frame collection and computation.

The authors' response has addressed my concerns, I keep my score.

Thank you for your response and for your positive evaluation of our work! We are glad that our rebuttal addressed your concerns, including the ablation on the benefits of geometric features and the latency analysis.

We have uploaded a revised manuscript that includes the backbone ablation in Section 4.4 (Table 6) and the extended runtime and latency analysis in Appendix B.3 (Table 8).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response and summary of updates

We thank all reviewers for their valuable feedback. We appreciate their recognition of our idea to ground hand reconstruction in scene geometry as an interesting and practical alternative to conventional hand-centric pipelines. We are encouraged by their positive assessments of G4D-Hand’s strong hand localization and trajectory accuracy, computational efficiency, and robustness to occlusion.

Reviewers’ suggestions have motivated additional method ablations, evaluations on new benchmarks, and a more detailed analysis of inference latency and world-frame trajectories. We present and discuss these results in the individual responses, and changes will be highlighted in blue in the revised manuscript. The main findings and additions can be summarized as follows:

  • Benefits of geometric backbone features (94jC, 7w13). We compare G4D-Hand using the CUT3R backbone with a variant using a visual backbone of the same size (ViT-L), both with and without raymap conditioning, to assess the benefits of geometric features. On HOT3D, geometric features reduce trajectory and wrist errors by up to ~14% under the matched training protocol.

  • Strong performance without pretrained hand priors in MANO regression head (7w13, coiu). Our original model initializes the MANO regression head with HaMeR decoder weights, reinitializing and training only selected layers. We now include two new ablations that 1) fine-tune the full head and 2) train it from random initialization. Both achieve results close to the default configuration when trained for the same number of epochs, indicating that G4D-Hand’s strong performance does not require pretrained HaMeR decoder weights.

  • Extended world-frame evaluation (7w13, coiu). We extend the world-frame evaluation with comparisons using SLAM-estimated camera trajectories and results on longer temporal windows. G4D-Hand achieves the lowest errors among the evaluated methods when using SLAM poses, while G4D-Hand with native CUT3R poses remains better or comparable with HaWoR across tested window lengths.

  • Extended speed and streaming latency analysis (94jC). We report end-to-end streaming latency for camera- and world-frame hand trajectory estimation for G4D-Hand and six other methods on the same hardware. Among temporal methods, G4D-Hand achieves the highest FPS and lowest latency. For metric world frame estimation, G4D-Hand achieves 2.8x higher FPS than HaWoR (CVPR 2025), which runs offline, with an average streaming latency of 1.14 s versus 15.50 s on 128-frame clips.

  • Generalization to unseen datasets (coiu). We added evaluations on SHOW3D (CVPR26) and HoloAssist (ICCV23), neither of which is used to train G4D-Hand, with comparisons against the two most recent methods ReViV (ECCV26) and EgoForce (SIGGRAPH26). G4D-Hand achieves the lowest trajectory errors on both, even compared to ReViV, which is trained on HoloAssist.

In summary, our additional experiments show the benefits of geometric backbone features, indicate that G4D-Hand’s strong performance does not require initialization of the MANO regression head with HaMeR decoder weights, confirm generalization to unseen datasets, and extend the world-frame and runtime analysis. We hope these additions address the reviewers’ concerns and are happy to provide further clarification.

AuthorsOpenReview ↗

Revised manuscript uploaded

We have uploaded a revised manuscript that includes the experiments, analyses, and clarifications from our rebuttal responses. For easier reference, changes are highlighted in blue.

The main updates are:

  • Backbone ablation. Table 6 (Section 4.4) compares CUT3R with an ImageNet ViT-L backbone, with and without raymap conditioning.
  • MANO regression head initialization ablation. Table 6 (Section 4.4) compares our default setting with two alternatives: 1) fine-tuning the full MANO regression head and 2) re-initializing the head and training it from scratch.
  • Extended world-frame evaluation. Section 4.2 and Table 3 present evaluation over longer temporal windows. Appendix D.5 and Table 12 add comparisons with baselines using SLAM-estimated camera poses.
  • Extended runtime analysis. Appendix B.3 and Table 8 compare the speed and latency of G4D-Hand and six related works.
  • Generalization to unseen datasets. Appendix D.1 and Table 9 present evaluation on SHOW3D and HoloAssist. Figure 11 (Appendix E.2) visualizes the trajectories estimated by G4D-Hand and ReViV on SHOW3D.
  • Extended related work. Section 2 now includes CLIFF discussion and clarifies the differences between G4D-Hand and WildHands, CLIFF, and Human3R.
  • Additional baseline. WildHands is added to the camera-frame ARCTIC comparisons in Table 1 (Section 4.1) and Table 11 (Appendix D.3).

We thank the reviewers again for their valuable feedback, which has helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.