acceptodds
Under review as a conference paper at ICLR 2027

V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos

15 views

Abstract

Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present \method, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model achieves the lowest coverage-aware mean per-joint position error (MPJPE) among comparable baselines on ARCTIC, HOT3D, and EgoStandard. It also yields lower world-frame trajectory errors than the compared SLAM-based pipelines. Ablations support the benefits of video pretraining, temporal context, and mixed supervision. \method achieves a forward-pass throughput of  FPS on a single RTX A6000. We will publicly release our code, model weights, and training scripts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video, 4D Reconstruction, Hand Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AoegOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

V-JEPA Hand is a feed-forward video model that jointly recovers bilateral hand motion, hand presence, camera motion, and clip-level focal lengths from an egocentric clip in one forward pass, without a hand detector, crops, SLAM, or test-time optimization. A self-supervised V-JEPA 2.1 ViT-L encoder ingests space-time tubelets; a query-based ViT predictor takes the encoder grid and an intrinsics token as input, using one learnable query for each output frame. Per-frame outputs contain, per hand, camera-space wrist translation, 6D wrist rotation, 45 articulation Euler angles, and a presence logit, plus a 9D camera transform to the first-frame anchor with signed-log translation, and two focal-length predictions.

The model is evaluated on ARCTIC, HOT3D (HaWoR-aligned split), and EgoStandard. V-JEPA Hand achieves near-perfect detection coverage and the lowest coverage-aware MPJPE on all three, and the best world-frame W-MPJPE, WA-MPJPE, and RTE versus SLAM-based pipelines.

Critical strengths

(a) A compelling simplicity-speed-accuracy operating point combined with the right evaluation metric. The detector-free, SLAM-free, optimization-free design runs at >90 FPS while achieving the lowest coverage-aware MPJPE, which penalizes missed hands rather than scoring only easy true-positive frames. The detection advantage is large and consequential (e.g., WildHands recall 0.425 on HOT3D vs. 0.978 for the proposed model), indicating that much of prior methods' apparent accuracy relies on frame selection.

(b) Credible generalization evidence and thorough component ablations. EgoStandard is produced by an independent commercial pipeline and is completely unseen during training, yet the model still leads most metrics; the pseudo-label ablation is evaluated on that unseen benchmark and shows improved geometry rather than merely more detections, and the pretraining, temporal-context, and encoder ablations isolate the source of gains. This supports both the data recipe and the architecture.

Critical weaknesses

(a) The technical novelty reduces largely to an encoder swap, and the evidence that the swap is what helps is weak. The architecture — frozen video encoder plus query-token transformer regressing MANO parameters and camera pose — is essentially ViDiHand's recipe with V-JEPA 2.1 in place of the previous backbone. That is a legitimate finding if the paper isolates it, but the ablations run under a different training recipe and a different world horizon than the headline table (the paper says so itself), so the one experiment that would establish "V-JEPA 2.1 is the reason" is not comparable to the result it is meant to explain. Without that, the contribution is an engineering report on a strong backbone rather than a transferable insight.

(b) Circular pseudo-label supervision and under-documented train/test mismatch. The large-scale training labels are generated by HaWoR (and VITRA), which is also the main learned baseline, so the model and its chief comparator share an estimator family and no pseudo-label accuracy statistics are given; no limitations section is provided. Training uses 16-frame clips while scoring uses full-minute tracks, and the stitching of overlapping clip predictions into world-frame trajectories (and boundary effects) is not explained. Finally, fixing MANO shape at beta = 0 ignores identity-dependent hand geometry, which can bias meshes and contact analyses that motivate the application.

Questions for authors

(1) Why are StableHand, HandFlow, ReViV, and related 2026 systems not compared? Please provide head-to-head coverage-aware and world-frame results on the same splits.

(2) What is the accuracy of the HaWoR/VITRA pseudo-labels against mocap GT, and how large is the advantage over HaWoR when pseudo-labels are replaced by GT on a matched subset?

(3) How are 16-frame predictions stitched into the full tracks used in Tables 1-2 (window stride, coordinate alignment, smoothing), and what errors appear at clip boundaries? Can the model run natively on longer clips?

3 comments

Part 1: Methodological contribution and predictor controls

We thank the reviewer.

Our main model jointly fine-tunes the encoder and predictor; the frozen encoder is only an ablation. Unlike ViDiHand's adaptation through hand-overlay generation, we directly adapt predictive, self-supervised video representations through a single-stage reconstruction objective. Shared features jointly recover bilateral hand motion, hand presence and camera egomotion, producing world-coordinate hand trajectories without separate SLAM or test-time optimization.

We added two simplified predictor controls:

  • MLP regression head: spatially pools each temporal token group to 2-by-2 before regression, without head-level temporal attention.
  • One-block Query predictor: retains one joint attention block over video, intrinsic and frame-query tokens, versus 24 blocks in Main.

All three models jointly fine-tune V-JEPA 2.1 ViT-L with the same documented training settings and 30,519 optimization steps, and receive GT camera intrinsics at evaluation. Below are all 14 original hand metrics.

ARCTIC

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9960.9980.99915.5915.366.846.598.998.8512.4612.180.0320.0316.09
Query0.9970.9980.99915.0514.876.826.6310.3010.2011.5111.290.0320.0316.53
MLP0.9980.9990.99914.8014.676.906.7610.5910.5210.8310.670.0250.0248.85

HOT3D

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9710.9790.98014.4212.048.245.7216.815.719.906.520.0260.0145.74
Query0.9710.9790.98014.6812.318.365.8517.306.1810.076.700.0280.0165.87
MLP0.9700.9780.98015.0812.648.545.9518.096.5510.376.930.0280.0179.15

EgoStandard

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9900.9950.99716.9716.308.017.2911.106.5110.609.840.0450.0434.88
Query0.9930.9960.99817.3216.877.957.4710.507.4310.9110.390.0520.0504.82
MLP0.9910.9950.99717.8617.268.237.5812.998.9411.2510.570.0540.0537.70

Main achieves lower PA-MPJPE and EPE than both simplified heads on all three datasets, and lower Jitter than MLP. Even MLP substantially outperforms WiLoR, HaMeR, WildHands and HaWoR in MPJPE-p across all three datasets. These results support the value of reconstruction-adapted predictive video representations, with further benefits from the full predictor, rather than decoder depth alone.

The transferable insight is a predictive-representation route to joint hand-camera reconstruction. We combine large-scale pseudo-labelled video with a comparatively small amount of high-quality laboratory annotations in one training stage. Strong performance on EgoStandard, excluded from reconstruction training, demonstrates generalization beyond the training domains.

Encoder ablation. Table 6 compares pretrained ViT-L encoders using the same predictor, fine-tuning recipe and evaluation protocol, with eight temporal token groups aligned to make computational budgets as comparable as possible. V-JEPA 2.1 achieves the lowest MPJPE-p, W-MPJPE and WA-MPJPE on both ARCTIC and HOT3D, supporting our encoder choice. Although this ablation uses a different training setup and HOT3D split from the main table, all encoders within Table 6 share those conditions. We will clarify this scope.

Part 2: Comparisons with recent methods

We add ACE-Ego-Hand, ReViV and HandFlow on the paper's test splits and 81-frame evaluation protocol. The tables report all 14 original hand metrics.

ARCTIC

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9910.9950.99717.7217.088.217.5110.329.8613.0112.260.0200.0172.67
ReViV0.9220.9530.96238.0833.3219.7214.0473.6642.5932.9127.680.0930.0675.57
HandFlow0.8260.8920.94245.6035.9822.259.8095.0530.4033.1220.910.2390.1959.30
Ours0.9960.9980.99915.5915.366.846.598.998.8512.4612.180.0320.0316.09

HOT3D

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9460.9590.97617.1112.0611.275.9722.766.2412.345.860.0480.0272.83
ReViV0.8340.9010.90653.3444.1225.5313.2782.0824.6847.4035.630.1070.0646.90
HandFlow0.7750.8500.91033.3215.3725.956.7184.367.7330.509.940.1480.0858.86
Ours0.9710.9790.98014.4212.048.245.7216.815.719.906.520.0260.0145.74

EgoStandard

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9910.9950.99721.3520.799.548.9213.899.6513.8813.220.1570.1561.99
ReViV0.1110.2520.277109.7646.54102.1116.13642.9533.00125.5143.180.3310.1459.39
HandFlow0.9230.9540.97429.6224.5814.698.9346.588.2618.2511.970.1270.1138.22
Ours0.9900.9950.99716.9716.308.017.2911.106.5110.609.840.0450.0434.88

Ours achieves the lowest MPJPE-p, MPJPE, PA-MPJPE-p and PA-MPJPE on all three datasets.

World-coordinate comparisons. ACE-Ego-Hand does not estimate a camera trajectory and is therefore excluded. ReViV, HandFlow and Ours use the same common-TP hand-frame support within each dataset. W-MPJPE/WA-MPJPE are in mm; RTE is in percent.

DatasetMethodW-MPJPE↓WA-MPJPE↓RTE↓
ARCTICReViV139.5367.1159.40
ARCTICHandFlow73.7240.9938.04
ARCTICOurs34.7319.6517.85
HOT3DReViV123.5947.4653.95
HOT3DHandFlow60.1827.7635.25
HOT3DOurs33.4215.9925.38
EgoStandardReViV127.4250.5954.71
EgoStandardHandFlow66.4337.3644.40
EgoStandardOurs48.0824.8030.85

Ours also achieves the lowest W-MPJPE, WA-MPJPE and RTE on all three datasets, supporting the effectiveness of joint hand-camera recovery against these recent methods.

StableHand. StableHand releases pretrained models and precomputed inputs for selected clips, but not the complete preprocessing pipeline needed to evaluate new videos. We therefore could not complete its evaluation on our test split.

Part 3: Supervision, temporal stitching and shape

Pseudo-label supervision and generalization. We reuse VITRA's released Ego4D annotations and use the VITRA labeling pipeline for in-house videos, together with high-quality ARCTIC/HOT3D annotations.

As shown in Table 3 of the paper, the training-data ablation fixes architecture, initialization and training budget. Adding pseudo-labelled videos improves EgoStandard MPJPE-p from 18.68 to 17.44 mm and W-MPJPE from 70.95 to 64.12 mm, despite slightly lower recall (0.998 to 0.996). Neither model trains on EgoStandard. The additional data improves unseen-domain geometry, not merely detection coverage.

The advantage is not solely due to teacher-generated supervision. The GT-only model in Table 3, trained on ARCTIC/HOT3D without Ego4D or in-house pseudo-labels, achieves 18.68 mm MPJPE-p on EgoStandard versus HaWoR's 33.38 mm on the same benchmark (Table 1), a 44% reduction. This shows that strong performance does not require reconstruction training on HaWoR-generated labels. This is not a same-image label-replacement experiment, however, and the released Ego4D annotations lack paired mocap GT for direct label-error measurement. We will acknowledge possible teacher bias as a limitation.

How 16-frame predictions are stitched. The reported world-evaluation units are 81-frame segments, not continuous minute-long trajectories. Main windows start at zero-based indices [0, 16, 32, 48, 64, 65]; later windows own overlapping hand predictions. Camera-only bridge windows start at [15, 31, 47, 63] and do not replace hand outputs.

Each local camera trajectory is normalized to its first camera and connected through shared predicted camera frames. A camera-only bridge is used when adjacent main windows have no overlapping frames. The final two main windows (64-79 and 65-80) overlap and are connected directly. This places the predictions in one camera coordinate system for the segment. No per-window GT alignment is used for stitching, and no temporal smoothing is applied to the reported hands.

World-coordinate performance after stitching. The results in Part 2 are computed on the assembled 81-frame sequences. Under the shared-support protocol, our method achieves lower W-MPJPE, WA-MPJPE and RTE than ReViV and HandFlow on all three datasets. These results support reconstruction across multiple inference windows within the evaluated 81-frame horizon.

MANO shape. Our training data already include MANO beta annotations. Predicting shape is a straightforward extension of our token-based framework: we can add a dedicated shape token and a lightweight regression head, supervised by the available beta annotations. The reported model fixes beta = 0, but this is a choice in the current implementation rather than a restriction of the framework.

Reviewer_LzuEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts a pretrained V-JEPA 2.1 encoder with a query-based predictor to jointly estimate hands and camera motion from egocentric video. It reports strong results on ARCTIC, HOT3D and held-out EgoStandard, together with ablations and inference-speed measurements.

Critical strengths
  • The results are empirically strong across three datasets, including held-out EgoStandard. Improvements in both coverage-aware hand accuracy and world-coordinate metrics make the approach practically interesting.
  • The architecture is simple: a shared video encoder and query-based predictor jointly recover hands and camera motion. The comparison with ViDiHand is useful, and the different HOT3D split is clearly acknowledged.
Critical weaknesses
  • The technical novelty appears limited. The method mainly adapts an existing video encoder using a relatively standard prediction architecture. Simplicity is a strength, but it remains unclear what methodological insight, beyond the effectiveness of pretrained video features, distinguishes this work.
  • The comparison omits several closely related recent methods, particularly ReViV, StableHand and ACE-Ego-Hand. ReViV is especially relevant to joint hand/camera prediction; the others are relevant to temporal hand recovery. Without comparisons under compatible protocols, or a clear explanation of availability constraints, the empirical advantage is difficult to place.
Questions for authors
  • The encoder uses two-frame tubelets and 16×16 spatial patches, producing a T/2 × H/16 × W/16 token grid. How well does this preserve fine finger articulation and rapid motion, especially when hands occupy only a small image region? Do you have results stratified by hand size or motion speed, or a resolution/tokenization ablation? Coarse tokenization alone does not establish that fine detail is lost.
3 comments

Part 1: Comparisons with recent methods

We thank the reviewer. We add comparisons with ACE-Ego-Hand, ReViV, and HandFlow using the paper's evaluation splits, protocol, and metrics.

StableHand availability. StableHand releases pretrained models and precomputed inputs for selected clips, but not the complete preprocessing pipeline needed to evaluate new videos. We therefore could not complete its evaluation on our test split.

ARCTIC

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EP-p↓EP↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9910.9950.99717.7217.088.217.5110.329.8613.0112.260.0200.0172.67
ReViV0.9220.9530.96238.0833.3219.7214.0473.6642.5932.9127.680.0930.0675.57
HandFlow0.8260.8920.94245.6035.9822.259.8095.0530.4033.1220.910.2390.1959.30
Ours0.9960.9980.99915.5915.366.846.598.998.8512.4612.180.0320.0316.09

HOT3D

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EP-p↓EP↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9460.9590.97617.1112.0611.275.9722.766.2412.345.860.0480.0272.83
ReViV0.8340.9010.90653.3444.1225.5313.2782.0824.6847.4035.630.1070.0646.90
HandFlow0.7750.8500.91033.3215.3725.956.7184.367.7330.509.940.1480.0858.86
Ours0.9710.9790.98014.4212.048.245.7216.815.719.906.520.0260.0145.74

EgoStandard

MethodAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EP-p↓EP↓GO-p↓GO↓CT-p↓CT↓Jit↓
ACE-Ego-Hand0.9910.9950.99721.3520.799.548.9213.899.6513.8813.220.1570.1561.99
ReViV0.1110.2520.277109.7646.54102.1116.13642.9533.00125.5143.180.3310.1459.39
HandFlow0.9230.9540.97429.6224.5814.698.9346.588.2618.2511.970.1270.1138.22
Ours0.9900.9950.99716.9716.308.017.2911.106.5110.609.840.0450.0434.88

Ours achieves lower MP-p and PA-p than ACE-Ego-Hand and HandFlow on all three datasets, while ACE-Ego-Hand achieves lower Jitter.

World-coordinate comparisons

All three methods use the same common-TP hand-frame support within each dataset. W-MPJPE/WA-MPJPE are in mm and RTE is in percent.

DatasetMethodW-MPJPE↓WA-MPJPE↓RTE↓
ARCTICReViV139.5367.1159.40
ARCTICHandFlow73.7240.9938.04
ARCTICOurs34.7319.6517.85
HOT3DReViV123.5947.4653.95
HOT3DHandFlow60.1827.7635.25
HOT3DOurs33.4215.9925.38
EgoStandardReViV127.4250.5954.71
EgoStandardHandFlow66.4337.3644.40
EgoStandardOurs48.0824.8030.85

Part 2: Methodological insight and contributions

Our central finding is that predictive, self-supervised video representations provide an effective alternative to video-generative priors for joint hand-camera reconstruction.

Predictive video representations with direct reconstruction supervision. ViDiHand demonstrates the potential of pretrained video diffusion representations, adapted through a hand-overlay rendering objective. We establish a different route: directly adapting V-JEPA 2.1 and its predictor through a single-stage reconstruction objective, without a hand-overlay generation stage. Our results show that predictive video representations can support fine-grained hand reconstruction after end-to-end adaptation.

Joint hand and camera recovery from shared features. Our model jointly predicts bilateral hand motion, hand presence, and camera egomotion from a unified spatiotemporal representation. Jointly estimating camera motion allows us to express hand trajectories in world coordinates, without a separate SLAM system or test-time optimization.

Practical adaptation and generalization. We combine large-scale pseudo-labelled video with a comparatively small amount of high-quality laboratory annotations in one training stage and evaluate the same trained model across ARCTIC, HOT3D, and EgoStandard, with EgoStandard excluded from reconstruction training. Together with the paper's accuracy and inference-efficiency results, these experiments demonstrate a simple, effective route to joint hand-camera recovery that transfers beyond the training domains.

Part 3: Performance on small hands and rapid motion

We evaluate robustness to hand size and hand motion speed, using the same checkpoint on all three datasets. Within each dataset, GT annotations define small/medium/large hands and slow/medium/fast hand motion. The tables show how reconstruction accuracy and hand recall vary across these groups.

1. How does performance change when hands are small?

Size groups use projected GT hand-box area relative to image area. Each tubelet covers 16x16 input pixels across two frames. The small-hand group averages approximately 2.9 by 3.3 patch widths at the model input. Each cell lists small / medium / large.

MetricARCTICHOT3DEgoStandard
Rec (%)↑99.41 / 99.99 / 100.0095.26 / 98.85 / 99.5798.71 / 99.74 / 99.91
MP-p↓15.94 / 14.80 / 16.0518.56 / 12.68 / 12.0317.87 / 16.33 / 16.72
MP↓15.25 / 14.79 / 16.0513.40 / 11.27 / 11.5116.28 / 16.01 / 16.62
PA-p↓7.73 / 6.32 / 6.4811.96 / 7.04 / 5.729.16 / 7.48 / 7.38
PA↓6.99 / 6.31 / 6.486.47 / 5.56 / 5.177.45 / 7.14 / 7.27
EP-p↓7.74 / 7.59 / 11.4629.53 / 12.40 / 9.3917.08 / 8.27 / 7.96
EP↓7.36 / 7.54 / 11.464.84 / 5.21 / 6.986.20 / 6.12 / 7.21
GO-p↓13.19 / 12.23 / 11.9615.34 / 8.07 / 6.2811.92 / 9.65 / 10.24
GO↓12.36 / 12.22 / 11.967.77 / 6.18 / 5.6710.04 / 9.35 / 10.12
CT-p↓0.036 / 0.030 / 0.0310.045 / 0.020 / 0.0130.056 / 0.042 / 0.035
CT↓0.032 / 0.030 / 0.0310.019 / 0.013 / 0.0120.052 / 0.042 / 0.035
Jit↓6.43 / 6.04 / 5.826.95 / 5.59 / 4.765.43 / 4.77 / 4.44

Matched-hand pose accuracy remains similar across the tested hand sizes, including small-hand groups averaging only a few spatial patch widths.

2. Does fast motion consistently degrade reconstruction?

Speed is the mean 3D displacement of the 21 GT joints per unit time on consecutive source frames, measured in camera coordinates (including camera motion). The fast group averages approximately 0.47 m/s across datasets. Each cell lists slow / medium / fast.

MetricARCTICHOT3DEgoStandard
Rec (%)↑99.83 / 99.87 / 99.7097.04 / 97.83 / 98.8299.72 / 99.50 / 99.20
MP-p↓15.55 / 15.65 / 15.5816.05 / 14.25 / 12.9416.55 / 16.79 / 17.48
MP↓15.34 / 15.50 / 15.2312.71 / 11.84 / 11.5516.20 / 16.18 / 16.49
PA-p↓6.89 / 6.86 / 6.789.41 / 8.30 / 7.007.67 / 8.03 / 8.23
PA↓6.67 / 6.70 / 6.405.87 / 5.76 / 5.547.30 / 7.38 / 7.16
EP-p↓8.73 / 9.23 / 9.0022.75 / 17.29 / 10.368.77 / 10.58 / 13.44
EP↓8.65 / 9.11 / 8.786.47 / 5.48 / 5.176.41 / 6.41 / 6.69
GO-p↓12.17 / 12.36 / 12.8511.80 / 9.89 / 7.9910.20 / 10.32 / 11.16
GO↓11.92 / 12.18 / 12.437.06 / 6.40 / 6.119.77 / 9.63 / 10.06
CT-p↓0.033 / 0.031 / 0.0330.030 / 0.026 / 0.0210.044 / 0.045 / 0.045
CT↓0.032 / 0.030 / 0.0310.014 / 0.014 / 0.0150.043 / 0.043 / 0.042
Jit↓4.63 / 5.68 / 8.004.44 / 5.24 / 7.554.00 / 4.59 / 6.06

Hand-pose accuracy and recall remain stable across the evaluated motion-speed groups, with recall above 97% in every group. Higher Jitter in the fast-motion groups may partly reflect the underlying motion itself: even a prediction that exactly follows GT can have high Jitter when hand velocity changes rapidly. Thus, higher Jitter alone does not imply larger reconstruction error.

Together, these results support robust hand-pose recovery over the hand-size and motion-speed ranges observed in these egocentric benchmarks.

Our contribution is a general hand-camera reconstruction framework; the resolution and sampling rate used here define one evaluation setting, not the scope of the formulation. To address even smaller hands and faster motion, the same framework can accommodate higher spatial and temporal resolutions without changing its core joint hand-camera formulation.

Reviewer_5iPQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors proposed V-JEPA Hand, a simple encoder-predictor video model with no detector, crops or SLAM. Instead, it uses V-JEPA 2.1 to recover hand and camera motion from egocentric videos. The model is trained using pseudo-labelled Ego4D and in-house videos plus ARCTIC and HOT3D. It performs well on three benchmarks, including unseen configuration, EgoStandard.

Critical strengths
  • Authors introduce a simple feed-forward video model that achieves a promising throughput of 91.3 FPS on a single GPU.

  • The proposed simple method achieves comparable performance to prior methods on three benchmarks: ARCTIC, HOT3D, and EgoStandard.

Critical weaknesses
  • A basic V-JEPA baseline is missing: I would expect to see a simple regression head trained under the same conditions. This would show what the proposed predictor adds beyond the backbone.

  • Less clear technical contribution: Most of the improvement seemingly comes from the V-JEPA encoder by producing meaningful features in latent space. The claim that video diffusion models are too heavy is also weakened by DreamHand, which runs a single-pass diffusion encoder at 63 FPS. I think more backbone and predictor changes are necessary to properly explain the technical contribution.

  • Experimental results and settings need more explanation: Camera trajectory and intrinsics are not evaluated directly, despite being in the title. No ATE, RPE, or camera rotation errors are reported. Focal-length prediction, listed as a contribution, is never evaluated. It is also unclear whether V-JEPA Hand and the baselines received ground-truth focal lengths at test time. If only V-JEPA Hand did, the EPE and translation comparisons are biased. Table 6 seemingly mixes several factors. For example, V-JEPA uses resolution 384, while the alternatives use 224. Frame counts and pretraining data also differ. These differences make it hard to explain V-JEPA’s advantage. Need a more similar scale comparison for demonstrating V-JEPA's design choice.

  • Unclear data usage: The pseudo-label pipeline requires more description and the in-house data lack basic details such as recording duration, participant counts, cameras, scene composition, etc. The data release plan is also unclear. They are necessary for reproducibility.

Questions for authors
  • What is the main technical novelty?

  • Which models and checkpoints generate the pseudo-labels? How are unreliable labels filtered? These choices may affect label quality and are needed to reproduce training.

5 comments

Part 1: Simple-head controls and technical contribution

We added two simplified predictor baselines: a pooled MLP regression head and a one-block query predictor. We also evaluate camera motion and focal-length prediction.

In the MLP setting, each temporal token group is spatially pooled to 2-by-2 before regression, without head-level temporal attention. In the Query setting, one joint attention block operates over video, intrinsic and frame-query tokens, compared with 24 blocks in the main predictor.

These two variants use the same documented training settings and number of optimization steps as the main model. All variants jointly fine-tune V-JEPA 2.1 ViT-L and receive GT camera intrinsics as input at evaluation.

We report all 14 original hand metrics.

ARCTIC

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9960.9980.99915.5915.366.846.598.998.8512.4612.180.0320.0316.09
Query0.9970.9980.99915.0514.876.826.6310.3010.2011.5111.290.0320.0316.53
MLP0.9980.9990.99914.8014.676.906.7610.5910.5210.8310.670.0250.0248.85

HOT3D

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9710.9790.98014.4212.048.245.7216.815.719.906.520.0260.0145.74
Query0.9710.9790.98014.6812.318.365.8517.306.1810.076.700.0280.0165.87
MLP0.9700.9780.98015.0812.648.545.9518.096.5510.376.930.0280.0179.15

EgoStandard

HeadAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
Main0.9900.9950.99716.9716.308.017.2911.106.5110.609.840.0450.0434.88
Query0.9930.9960.99817.3216.877.957.4710.507.4310.9110.390.0520.0504.82
MLP0.9910.9950.99717.8617.268.237.5812.998.9411.2510.570.0540.0537.70

The full predictor provides consistent gains over both simplified heads. Across all three datasets, it achieves lower PA-MPJPE and EPE than both the one-block query predictor and the pooled MLP head. It also achieves the lowest EPE-p on ARCTIC and HOT3D, and lower Jitter than the MLP head on all three datasets. Thus, the full predictor improves reconstruction quality, although it is not best on every metric.

Even the pooled MLP substantially outperforms WiLoR, HaMeR, WildHands and HaWoR in MPJPE-p across all three datasets. This suggests that a substantial part of the gains comes from the strong V-JEPA 2.1 representations adapted through reconstruction training, while the main predictor provides further improvements.

Our contribution is an effective predictive-representation approach to joint hand-camera reconstruction. We show that predictive, self-supervised video representations offer an alternative to video-generative priors. We directly adapt V-JEPA 2.1 and its predictor with a single-stage reconstruction objective. Shared spatiotemporal features support joint recovery of bilateral hands, hand presence and camera motion, enabling world-coordinate hand trajectories without separate SLAM or test-time optimization. For practical adaptation and generalization, we combine large-scale pseudo-labelled video with a comparatively small amount of high-quality laboratory annotations in one training stage. Our strong performance on EgoStandard, which is excluded from reconstruction training, demonstrates robust generalization beyond the training domains.

Part 2: Camera, Intrinsics, and Pseudo-label generation

Direct camera comparisons with baselines.

We compare camera trajectories and world-coordinate hand reconstruction with the original world baselines, using shared evaluation support for each metric within each dataset.

DatasetMethodATE (mm)↓RPE-T (mm)↓RPE-R (deg)↓Camera rotation error (deg)↓W-MPJPE (mm)↓WA-MPJPE (mm)↓RTE (%)↓
ARCTICWiLoR+SLAM19.963.370.171.3263.3634.1329.78
ARCTICHaMeR+SLAM19.963.370.171.3269.0533.7828.21
ARCTICHaWoR52.677.350.131.14106.0555.3848.92
ARCTICOurs9.782.070.221.1635.8619.7217.58
HOT3DWiLoR+SLAM18.371.790.070.3074.7828.4240.21
HOT3DHaMeR+SLAM18.371.790.070.30105.1035.0847.31
HOT3DHaWoR17.232.490.060.2863.0827.6051.02
HOT3DOurs11.772.760.271.6634.0816.1828.15
EgoStandardWiLoR+SLAM35.043.880.331.9990.6741.5426.27
EgoStandardHaMeR+SLAM35.043.880.331.9994.8642.7427.16
EgoStandardHaWoR33.885.400.301.7390.2345.0235.93
EgoStandardOurs27.884.020.351.9766.3532.9122.37

ATE measures camera-position error after rigid trajectory alignment. RPE-T/RPE-R measure relative translation/rotation errors between adjacent evaluated frames; camera rotation error measures orientation error after first-frame alignment. W-MPJPE/WA-MPJPE measure world-coordinate joint errors after first-frame/whole-track alignment. RTE is whole-track-aligned wrist error normalized by the GT wrist's maximum displacement from its starting point. All alignments exclude scale fitting.

Our method achieves the lowest ATE, W-MPJPE, WA-MPJPE and RTE on all three datasets. This shows consistent gains in global camera trajectory accuracy and world-coordinate hand reconstruction over the compared baselines, although relative-motion and rotation errors are not uniformly better.

Test-time camera intrinsics.

Our main evaluation supplies the ground-truth camera intrinsic matrix K. We compare the same checkpoint with GT intrinsics supplied versus withheld, reporting all 14 hand metrics below. When K is withheld, the model predicts K itself and uses it to project the reconstructed 3D joints into the image.

WiLoR, HaMeR, WildHands and HaWoR were also evaluated using ground-truth camera intrinsics.

DatasetKAcc↑Rec↑F1↑MP-p↓MP↓PA-p↓PA↓EPE-p↓EPE↓GO-p↓GO↓CT-p↓CT↓Jit↓
ARCTICGiven0.9960.9980.99915.5915.366.846.598.998.8512.4612.180.0320.0316.09
ARCTICWithheld0.9960.9980.99915.5615.326.846.598.948.8112.4412.160.0320.0306.29
HOT3DGiven0.9710.9790.98014.4212.048.245.7216.815.719.906.520.0260.0145.74
HOT3DWithheld0.9710.9790.98014.4312.058.235.7217.246.169.906.530.0270.0155.76
EgoStandardGiven0.9900.9950.99716.9716.308.017.2911.106.5110.609.840.0450.0434.88
EgoStandardWithheld0.9890.9940.99717.0816.378.067.3011.536.6910.659.840.0560.0545.24

Most hand metrics change little without GT intrinsics. With K withheld, our MPJPE-p remains 26%, 50% and 49% lower than the best of WiLoR, HaMeR, WildHands and HaWoR on ARCTIC, HOT3D and EgoStandard, respectively. Thus, predicted intrinsics support accurate hand reconstruction while retaining a substantial advantage over these baselines.

Pseudo-label generation and in-house data.

For Ego4D, we reuse VITRA's released pseudo-labels rather than regenerate them. VITRA uses HaWoR for hand reconstruction (without motion infilling) and MegaSAM with MoGe-2 depth for metric camera poses; intrinsics are estimated with DroidCalib, DeepCalib or MoGe-2. Our in-house videos follow the VITRA labeling pipeline. Our in-house training data comprise over 10,000 hours of videos covering diverse everyday scenarios.

Part 3: DreamHand (ACE-Ego-Hand) and encoder comparison

DreamHand clarification. DreamHand refers to ACE-Ego-Hand, which extracts diffusion-derived features in a single pass. We will revise the manuscript to clarify that using diffusion-derived features does not necessarily require multiple denoising steps.

Encoder ablation. We designed Table 6 to make the comparison as fair and controlled as possible while respecting each encoder's image/video processing structure. We use the same ViT-L model scale, predictor design and fine-tuning recipe, and align the temporal token budget at eight groups: 16 frames with two-frame tubelets for video encoders versus eight independently encoded frames for image encoders. These choices balance a comparable temporal token budget with encoder-specific input requirements. We will make these controls and input settings explicit in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.