- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes G4D-Hand, a method for 4D hand trajectory estimation from egocentric video. It is built on a frozen 4D reconstruction backbone, CUT3R. Instead of detecting and cropping hands, left- and right-hand branches extract hand features directly from full-frame CUT3R features, which are conditioned on camera intrinsics through a raymap. A temporal transformer refines these features. A HaMeR-initialized decoder then predicts MANO pose and shape, and a separate head predicts metric wrist translation from CUT3R’s feature pyramid. A second CUT3R pass estimates the camera motion, which lifts the hands to the world frame without external SLAM. The model is trained on HOT3D, ARCTIC, and H2O and evaluated on the same three datasets. It achieves the lowest trajectory, wrist, and inter-hand errors, with articulation accuracy comparable to the state of the art.
Critical strengths
- Clear idea and simple design. The paper tests a clear hypothesis: hand reconstruction should be grounded in scene geometry. It estimates both the hands and the camera with a single frozen CUT3R backbone, without a hand detector, a hand-specific image encoder, or external SLAM.
- Large gains in localization and trajectory accuracy. On all three benchmarks, the method achieves the lowest trajectory error (G-MPJPE), wrist error, and inter-hand error (MRRPE). On ARCTIC, for example, it reduces G-MPJPE by 31% and MRRPE by about 50% relative to the strongest baseline for each metric. Articulation accuracy (RA-MPJPE) remains comparable to the best existing methods.
- Robustness to occlusion. Appendix D.3 shows that the method has the lowest wrist error and MRRPE in every occlusion bin on ARCTIC, including when 75–100% of the joints are occluded.
- Efficiency. The model trains only 27M parameters, and camera-frame prediction runs at over 30 FPS on a single RTX 4090.
Critical weaknesses
- Possible advantage from in-domain training. G4D-Hand is trained only on the three evaluation datasets (HOT3D, ARCTIC, and H2O). The baselines are evaluated with their public checkpoints and are not retrained on the same data. Part of the gain may therefore come from in-domain training. There is also no evaluation on unseen datasets.
- No long-sequence evaluation with estimated cameras. The CUT3R state is reset every 16 frames, or every 64 frames for camera estimation. World-frame results with estimated camera motion cover only 64-frame clips. Full-sequence results (Table 8) are reported only with ground-truth camera poses, so the full pipeline is not verified on long videos.
- Unclear handling of invisible hands. The method uses no hand detector, but the paper does not explain what it outputs when a hand is out of view or only one hand is visible, or whether it predicts hand presence. The evaluation also covers only in-view hands (Appendix C.1).
- Limited novelty.
- Because the reference pose is fixed, the raymap only encodes camera intrinsics. This serves a similar purpose to the intrinsics inputs in CLIFF and WildHands, and the paper does not compare against these alternatives.
- The method relies on the pretrained HaMeR MANO decoder, and the paper does not analyze how much this hand-specific prior contributes.
- Conceptually, it is close to Human3R, which also builds on CUT3R. The main difference is that G4D-Hand removes the hand-specific feature encoder (Human3R uses Multi-HMR features).
Questions for authors
- G4D-Hand is trained only on HOT3D, ARCTIC, and H2O, while the baselines are evaluated with their public checkpoints. How much of the improvement comes from this in-domain training? Can the authors report (a) results of strong baselines (e.g., WiLoR) fine-tuned on the same data, or (b) results of G4D-Hand on a dataset not used for training (e.g., OakInk2 or HOI4D)? (could change my assessment)
- World-frame results with estimated cameras are limited to 64-frame clips. How does the full pipeline perform on longer sequences? How are 64-frame chunks, each with its own CUT3R state and scale estimate, connected into a full trajectory? (could change my assessment)
- What does the model output when a hand is out of view or only one hand is visible? Does it predict hand presence? How are such frames handled in training? Evaluation covers only in-view hands (Appendix C.1). (clarification)
- The raymap uses a fixed reference pose, so it encodes only the camera intrinsics. Have the authors compared it with simpler ways of providing intrinsics, such as CLIFF- or WildHands-style inputs? (clarification)
- How much does performance depend on the pretrained HaMeR MANO decoder (e.g., compared with a randomly initialized decoder)? Beyond removing the hand-specific encoder, what is the main technical difference from Human3R? (clarification)
4 comments
Thank you for the thorough response. The new results on SHOW3D and HoloAssist address my concern about in-domain training, and the MANO-head ablation shows that the HaMeR initialization is not the main source of the gains. The clarification on out-of-view hands is helpful; please state this limitation explicitly in the paper. Two points remain partly open: for longer windows, the native pipeline performs on par with HaWoR at 192–240 frames, and the best results rely on external SLAM, which should be clarified given the paper’s no-SLAM claim; and the comparison with WildHands should include the numbers and, ideally, a controlled ablation replacing the raymap with CLIFF-style inputs. Overall, my concerns are largely resolved, and I keep my positive assessment.
Thank you for the thorough response. The new results on SHOW3D and HoloAssist address my concern about in-domain training, and the MANO-head ablation shows that the HaMeR initialization is not the main source of the gains. The clarification on out-of-view hands is helpful. Please state this limitation explicitly in the paper. Two points remain partly open: for longer windows, the native pipeline performs on par with HaWoR at 192–240 frames, and the best results rely on external SLAM, which should be clarified given the paper’s no-SLAM claim. In addition, the comparison with WildHands should include the numbers and, ideally, a controlled ablation replacing the raymap with CLIFF-style inputs. Overall, my concerns are largely resolved, and I keep my positive assessment.
Thank you for your response and positive evaluation of our work! We are happy that our rebuttal largely resolved your concerns, including the evaluation on unseen datasets and the MANO head initialization ablation.
We have uploaded a revised manuscript. We expanded the related work (Section 2) with a more detailed comparison to WildHands, CLIFF, and Human3R. We added quantitative evaluations on datasets not used during training in Appendix D.1 (Table 9) and qualitative examples in Appendix E.2 (Figure 11). We also added the world frame evaluation over longer temporal windows in Section 4.2 (Table 3). We included the MANO head initialization ablation in Section 4.4 (Table 6). In the limitations (Section 5), we added hand presence prediction to our future work, together with handling an arbitrary number of hands and out-of-view hands. We also included WildHands results on ARCTIC in Table 1 and Table 11.
Generalization to unseen datasets, extended world-frame evaluation, and robustness to hand prior initialization
We thank the reviewer for their valuable feedback and positive evaluation of our work. We appreciate their support for grounding hand reconstruction in scene geometry. We also thank the reviewer for highlighting our model’s efficiency, robustness to occlusion, and strong localization and trajectory accuracy as key strengths.
W1., Q1. Generalization to unseen datasets
We evaluate on two datasets unseen in training: SHOW3D (participant XYZ109 validation sequences, ~26k frames) and HoloAssist (152 randomly selected 128-frame clips). We compare G4D-Hand with ReViV (ECCV26) and EgoForce (SIGGRAPH26), both trained on at least HOT3D, H2O, and ARCTIC. These results support that G4D-Hand generalizes beyond the training datasets.
Table 1: Camera-frame evaluation. Best zero-shot results bolded. ReViV is trained on HoloAssist and is excluded from its zero-shot ranking.
G4D-Hand achieves the lowest trajectory errors on both datasets, reducing G-/GA-MPJPE by 29.4%/26.0% on HoloAssist and ~60% for both on SHOW3D.
W2., Q2. Longer-window world-frame evaluation
We extend the world-frame evaluation to 128-, 192- and 240-frame windows. For this experiment, CUT3R processes 128-frame chunks with a 16-frame overlap. Each chunk of camera poses is independently scaled to absolute metric with hand-to-scene depth ratio (Sec. 4.2), then rigidly aligned to its predecessor using shared camera poses, with overlapping poses blended. This recovers metric camera trajectories in the first chunk's reference frame without ground truth.
Table 2: World-frame evaluation on ARCTIC with estimated camera poses. G-/GA- stand for G-MPJPE and GA-MPJPE and suffixes denote window lengths. Best results bolded.
G4D-Hand with SLAM achieves the lowest errors across all temporal windows. Performance of G4D-Hand with (native) CUT3R camera poses is better or comparable with HaWoR, which uses adapted DROID-SLAM.
W3., Q3. Clarification on out-of-view hands
The model does not predict hand presence. Training uses 16-frame windows with both hands present (five or more 2D keypoints per hand within the frame). All models are evaluated on in-view hands, whether one or both are visible, consistent with H2O/ARCTIC protocols. We plan to explore scene-grounded hand presence prediction, out-of-view and more than two hand pose prediction in the future work.
W4., Q5. Influence of pretrained hand priors in MANO regression head
We analyze the influence of initialization of MANO regression head with weights from HaMeR MANO decoder with two controlled ablations. The results below suggest that this initialization is not the main driver of performance.
In the default G4D-Hand setting, the MANO regression head is initialized from HaMeR, with only selected parameters, mainly cross-attention layers, reinitialized and trained. Here, we compare it with two new settings: FT (fine-tuning the full head), and Reinit. from Scratch (training the head from random initialization). All variants use the same architecture and training data. We shorten training from 100 to 30 epochs due to rebuttal time and resource constraints.
Table 3: MANO regression head ablation evaluated in the camera frame on the HOT3D.
G4D-Hand with both ablation settings performs similarly to the default, with slightly higher MRRPE and wrist error and lower trajectory errors.
W4., Q4., Q5. Comparisons with related works
W4, Q4: Intrinsics conditioning. G4D-Hand conditions full-frame rather than crop features using CUT3R's pretrained raymap encoder to predict metric wrist translation. Both G4D-Hand and WildHands train on ARCTIC. On its validation set, ours achieves lower errors across all metrics, reducing wrist error by ~2x. We will add these results and discuss both works in the revision.
W4., Q5: Human3R. G4D-Hand keeps CUT3R frozen and uses raymap conditioning and a dedicated wrist regression head to predict metric wrist positions from its geometric features. Hand branches localize hands within full-frame features through learned queries, without a detector.