- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
V-JEPA Hand is a feed-forward video model that jointly recovers bilateral hand motion, hand presence, camera motion, and clip-level focal lengths from an egocentric clip in one forward pass, without a hand detector, crops, SLAM, or test-time optimization. A self-supervised V-JEPA 2.1 ViT-L encoder ingests space-time tubelets; a query-based ViT predictor takes the encoder grid and an intrinsics token as input, using one learnable query for each output frame. Per-frame outputs contain, per hand, camera-space wrist translation, 6D wrist rotation, 45 articulation Euler angles, and a presence logit, plus a 9D camera transform to the first-frame anchor with signed-log translation, and two focal-length predictions.
The model is evaluated on ARCTIC, HOT3D (HaWoR-aligned split), and EgoStandard. V-JEPA Hand achieves near-perfect detection coverage and the lowest coverage-aware MPJPE on all three, and the best world-frame W-MPJPE, WA-MPJPE, and RTE versus SLAM-based pipelines.
Critical strengths
(a) A compelling simplicity-speed-accuracy operating point combined with the right evaluation metric. The detector-free, SLAM-free, optimization-free design runs at >90 FPS while achieving the lowest coverage-aware MPJPE, which penalizes missed hands rather than scoring only easy true-positive frames. The detection advantage is large and consequential (e.g., WildHands recall 0.425 on HOT3D vs. 0.978 for the proposed model), indicating that much of prior methods' apparent accuracy relies on frame selection.
(b) Credible generalization evidence and thorough component ablations. EgoStandard is produced by an independent commercial pipeline and is completely unseen during training, yet the model still leads most metrics; the pseudo-label ablation is evaluated on that unseen benchmark and shows improved geometry rather than merely more detections, and the pretraining, temporal-context, and encoder ablations isolate the source of gains. This supports both the data recipe and the architecture.
Critical weaknesses
(a) The technical novelty reduces largely to an encoder swap, and the evidence that the swap is what helps is weak. The architecture — frozen video encoder plus query-token transformer regressing MANO parameters and camera pose — is essentially ViDiHand's recipe with V-JEPA 2.1 in place of the previous backbone. That is a legitimate finding if the paper isolates it, but the ablations run under a different training recipe and a different world horizon than the headline table (the paper says so itself), so the one experiment that would establish "V-JEPA 2.1 is the reason" is not comparable to the result it is meant to explain. Without that, the contribution is an engineering report on a strong backbone rather than a transferable insight.
(b) Circular pseudo-label supervision and under-documented train/test mismatch. The large-scale training labels are generated by HaWoR (and VITRA), which is also the main learned baseline, so the model and its chief comparator share an estimator family and no pseudo-label accuracy statistics are given; no limitations section is provided. Training uses 16-frame clips while scoring uses full-minute tracks, and the stitching of overlapping clip predictions into world-frame trajectories (and boundary effects) is not explained. Finally, fixing MANO shape at beta = 0 ignores identity-dependent hand geometry, which can bias meshes and contact analyses that motivate the application.
Questions for authors
(1) Why are StableHand, HandFlow, ReViV, and related 2026 systems not compared? Please provide head-to-head coverage-aware and world-frame results on the same splits.
(2) What is the accuracy of the HaWoR/VITRA pseudo-labels against mocap GT, and how large is the advantage over HaWoR when pseudo-labels are replaced by GT on a matched subset?
(3) How are 16-frame predictions stitched into the full tracks used in Tables 1-2 (window stride, coordinate alignment, smoothing), and what errors appear at clip boundaries? Can the model run natively on longer clips?
3 comments
Part 2: Comparisons with recent methods
We add ACE-Ego-Hand, ReViV and HandFlow on the paper's test splits and 81-frame evaluation protocol. The tables report all 14 original hand metrics.
ARCTIC
| Method | Acc↑ | Rec↑ | F1↑ | MP-p↓ | MP↓ | PA-p↓ | PA↓ | EPE-p↓ | EPE↓ | GO-p↓ | GO↓ | CT-p↓ | CT↓ | Jit↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACE-Ego-Hand | 0.991 | 0.995 | 0.997 | 17.72 | 17.08 | 8.21 | 7.51 | 10.32 | 9.86 | 13.01 | 12.26 | 0.020 | 0.017 | 2.67 |
| ReViV | 0.922 | 0.953 | 0.962 | 38.08 | 33.32 | 19.72 | 14.04 | 73.66 | 42.59 | 32.91 | 27.68 | 0.093 | 0.067 | 5.57 |
| HandFlow | 0.826 | 0.892 | 0.942 | 45.60 | 35.98 | 22.25 | 9.80 | 95.05 | 30.40 | 33.12 | 20.91 | 0.239 | 0.195 | 9.30 |
| Ours | 0.996 | 0.998 | 0.999 | 15.59 | 15.36 | 6.84 | 6.59 | 8.99 | 8.85 | 12.46 | 12.18 | 0.032 | 0.031 | 6.09 |
HOT3D
| Method | Acc↑ | Rec↑ | F1↑ | MP-p↓ | MP↓ | PA-p↓ | PA↓ | EPE-p↓ | EPE↓ | GO-p↓ | GO↓ | CT-p↓ | CT↓ | Jit↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACE-Ego-Hand | 0.946 | 0.959 | 0.976 | 17.11 | 12.06 | 11.27 | 5.97 | 22.76 | 6.24 | 12.34 | 5.86 | 0.048 | 0.027 | 2.83 |
| ReViV | 0.834 | 0.901 | 0.906 | 53.34 | 44.12 | 25.53 | 13.27 | 82.08 | 24.68 | 47.40 | 35.63 | 0.107 | 0.064 | 6.90 |
| HandFlow | 0.775 | 0.850 | 0.910 | 33.32 | 15.37 | 25.95 | 6.71 | 84.36 | 7.73 | 30.50 | 9.94 | 0.148 | 0.085 | 8.86 |
| Ours | 0.971 | 0.979 | 0.980 | 14.42 | 12.04 | 8.24 | 5.72 | 16.81 | 5.71 | 9.90 | 6.52 | 0.026 | 0.014 | 5.74 |
EgoStandard
| Method | Acc↑ | Rec↑ | F1↑ | MP-p↓ | MP↓ | PA-p↓ | PA↓ | EPE-p↓ | EPE↓ | GO-p↓ | GO↓ | CT-p↓ | CT↓ | Jit↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACE-Ego-Hand | 0.991 | 0.995 | 0.997 | 21.35 | 20.79 | 9.54 | 8.92 | 13.89 | 9.65 | 13.88 | 13.22 | 0.157 | 0.156 | 1.99 |
| ReViV | 0.111 | 0.252 | 0.277 | 109.76 | 46.54 | 102.11 | 16.13 | 642.95 | 33.00 | 125.51 | 43.18 | 0.331 | 0.145 | 9.39 |
| HandFlow | 0.923 | 0.954 | 0.974 | 29.62 | 24.58 | 14.69 | 8.93 | 46.58 | 8.26 | 18.25 | 11.97 | 0.127 | 0.113 | 8.22 |
| Ours | 0.990 | 0.995 | 0.997 | 16.97 | 16.30 | 8.01 | 7.29 | 11.10 | 6.51 | 10.60 | 9.84 | 0.045 | 0.043 | 4.88 |
Ours achieves the lowest MPJPE-p, MPJPE, PA-MPJPE-p and PA-MPJPE on all three datasets.
World-coordinate comparisons. ACE-Ego-Hand does not estimate a camera trajectory and is therefore excluded. ReViV, HandFlow and Ours use the same common-TP hand-frame support within each dataset. W-MPJPE/WA-MPJPE are in mm; RTE is in percent.
| Dataset | Method | W-MPJPE↓ | WA-MPJPE↓ | RTE↓ |
|---|---|---|---|---|
| ARCTIC | ReViV | 139.53 | 67.11 | 59.40 |
| ARCTIC | HandFlow | 73.72 | 40.99 | 38.04 |
| ARCTIC | Ours | 34.73 | 19.65 | 17.85 |
| HOT3D | ReViV | 123.59 | 47.46 | 53.95 |
| HOT3D | HandFlow | 60.18 | 27.76 | 35.25 |
| HOT3D | Ours | 33.42 | 15.99 | 25.38 |
| EgoStandard | ReViV | 127.42 | 50.59 | 54.71 |
| EgoStandard | HandFlow | 66.43 | 37.36 | 44.40 |
| EgoStandard | Ours | 48.08 | 24.80 | 30.85 |
Ours also achieves the lowest W-MPJPE, WA-MPJPE and RTE on all three datasets, supporting the effectiveness of joint hand-camera recovery against these recent methods.
StableHand. StableHand releases pretrained models and precomputed inputs for selected clips, but not the complete preprocessing pipeline needed to evaluate new videos. We therefore could not complete its evaluation on our test split.
Part 3: Supervision, temporal stitching and shape
Pseudo-label supervision and generalization. We reuse VITRA's released Ego4D annotations and use the VITRA labeling pipeline for in-house videos, together with high-quality ARCTIC/HOT3D annotations.
As shown in Table 3 of the paper, the training-data ablation fixes architecture, initialization and training budget. Adding pseudo-labelled videos improves EgoStandard MPJPE-p from 18.68 to 17.44 mm and W-MPJPE from 70.95 to 64.12 mm, despite slightly lower recall (0.998 to 0.996). Neither model trains on EgoStandard. The additional data improves unseen-domain geometry, not merely detection coverage.
The advantage is not solely due to teacher-generated supervision. The GT-only model in Table 3, trained on ARCTIC/HOT3D without Ego4D or in-house pseudo-labels, achieves 18.68 mm MPJPE-p on EgoStandard versus HaWoR's 33.38 mm on the same benchmark (Table 1), a 44% reduction. This shows that strong performance does not require reconstruction training on HaWoR-generated labels. This is not a same-image label-replacement experiment, however, and the released Ego4D annotations lack paired mocap GT for direct label-error measurement. We will acknowledge possible teacher bias as a limitation.
How 16-frame predictions are stitched. The reported world-evaluation units are 81-frame segments, not continuous minute-long trajectories. Main windows start at zero-based indices [0, 16, 32, 48, 64, 65]; later windows own overlapping hand predictions. Camera-only bridge windows start at [15, 31, 47, 63] and do not replace hand outputs.
Each local camera trajectory is normalized to its first camera and connected through shared predicted camera frames. A camera-only bridge is used when adjacent main windows have no overlapping frames. The final two main windows (64-79 and 65-80) overlap and are connected directly. This places the predictions in one camera coordinate system for the segment. No per-window GT alignment is used for stitching, and no temporal smoothing is applied to the reported hands.
World-coordinate performance after stitching. The results in Part 2 are computed on the assembled 81-frame sequences. Under the shared-support protocol, our method achieves lower W-MPJPE, WA-MPJPE and RTE than ReViV and HandFlow on all three datasets. These results support reconstruction across multiple inference windows within the evaluated 81-frame horizon.
MANO shape. Our training data already include MANO beta annotations. Predicting shape is a straightforward extension of our token-based framework: we can add a dedicated shape token and a lightweight regression head, supervised by the available beta annotations. The reported model fixes beta = 0, but this is a choice in the current implementation rather than a restriction of the framework.
Part 1: Methodological contribution and predictor controls
We thank the reviewer.
Our main model jointly fine-tunes the encoder and predictor; the frozen encoder is only an ablation. Unlike ViDiHand's adaptation through hand-overlay generation, we directly adapt predictive, self-supervised video representations through a single-stage reconstruction objective. Shared features jointly recover bilateral hand motion, hand presence and camera egomotion, producing world-coordinate hand trajectories without separate SLAM or test-time optimization.
We added two simplified predictor controls:
All three models jointly fine-tune V-JEPA 2.1 ViT-L with the same documented training settings and 30,519 optimization steps, and receive GT camera intrinsics at evaluation. Below are all 14 original hand metrics.
ARCTIC
HOT3D
EgoStandard
Main achieves lower PA-MPJPE and EPE than both simplified heads on all three datasets, and lower Jitter than MLP. Even MLP substantially outperforms WiLoR, HaMeR, WildHands and HaWoR in MPJPE-p across all three datasets. These results support the value of reconstruction-adapted predictive video representations, with further benefits from the full predictor, rather than decoder depth alone.
The transferable insight is a predictive-representation route to joint hand-camera reconstruction. We combine large-scale pseudo-labelled video with a comparatively small amount of high-quality laboratory annotations in one training stage. Strong performance on EgoStandard, excluded from reconstruction training, demonstrates generalization beyond the training domains.
Encoder ablation. Table 6 compares pretrained ViT-L encoders using the same predictor, fine-tuning recipe and evaluation protocol, with eight temporal token groups aligned to make computational budgets as comparable as possible. V-JEPA 2.1 achieves the lowest MPJPE-p, W-MPJPE and WA-MPJPE on both ARCTIC and HOT3D, supporting our encoder choice. Although this ablation uses a different training setup and HOT3D split from the main table, all encoders within Table 6 share those conditions. We will clarify this scope.