acceptodds
Under review as a conference paper at ICLR 2027

DewHand: Direct Metric World-Space Hand Motion Reconstruction from Egocentric Videos

21 views

Abstract

Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present **DewHand**, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT- with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
egocentric vision, 3D hand reconstruction, hand motion reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_du7sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DewHand jointly generates metric world-space hand and camera trajectories from monocular egocentric video using geometric and hand-specific features with flow matching (Secs. 3.1–3.4). Experiments report improved rigid-aligned hand reconstruction on HOT3D and ARCTIC, with additional camera evaluation, cross-device transfer, and component ablations (Tables 1–4).

Critical strengths
  1. With predicted hand shape, DewHand reduces SE(3)-aligned WA-MPJPE relative to StableHand from 43.74 to 28.56 mm on the HOT3D StableHand split and from 78.63 to 36.35 mm on ARCTIC (Table 1), providing substantial evidence for improved metric world-space reconstruction without privileged shape input.
  2. The controlled ablations support the design choices: removing camera supervision raises WA-MPJPE from 35.94 to 42.36 mm, while replacing flow matching with same-architecture regression raises it to 45.13 mm (Table 4; lines 430–468).
Critical weaknesses
  1. The experiments establish the benefit of camera supervision, but do not isolate the contribution of coupling hand and camera prediction to the proposed solution to error propagation. Joint generation is a central motivation (p. 2, lines 80–103), and the architecture exchanges information among hand and camera tokens (p. 5, lines 231–237). Table 4(a), however, changes camera supervision: WA-MPJPE improves from 42.36 to 35.94 mm. This supports the authors' stated supervision benefit, but does not distinguish useful auxiliary supervision from a benefit of joint prediction over separate prediction pathways supplied with the same geometric features and supervision. The comparisons to differently constructed multi-stage systems in Table 1 do not resolve that distinction. Consequently, the complete system's accuracy gains are convincing, while the architectural explanation for those gains remains less secure. A matched factorized hand/camera predictor retaining the encoders, training data, and camera supervision would be a focused future test; any already available control or a narrower interpretation of the current ablation would address the immediate concern.
  2. The transfer results expose a substantial weakness in relative two-hand geometry despite improved rigid-aligned trajectory accuracy. On H2O, DewHand improves SE(3)-aligned WA-MPJPE over WiLoR + ViPE from 22.04 to 20.68 mm, but nearly doubles MRRPE-A from 23.76 to 46.17 mm; its Sim(3)-aligned WA-MPJPE is also worse (19.54 versus 14.80 mm). On TACO, MRRPE-A is 60.95 versus 55.82 mm despite better SE(3)-aligned WA-MPJPE (Table 3, p. 8). MRRPE-A measures inter-wrist displacement after a shared rigid alignment (p. 6, lines 311–314), making this a limitation of the stated common-world-frame reconstruction task, rather than an unrelated downstream requirement. The authors acknowledge mixed transfer results (lines 423–425), and HoloAssist improves across metrics, so this is not evidence of universal transfer failure. It does show that improved scale-preserving trajectory scores alone do not establish reliable bimanual spatial relationships across devices. Examining existing predictions by hand visibility and relative-wrist error, with explicit alignment and frame-selection definitions, would help determine the failure regime and the appropriate scope of the generalization claim.
Questions for authors
  1. [Clarification; Weakness 1] In the existing Table 4(a) implementation, does “w/o camera supervision” remove the camera components from Lfm in Eq. (2) as well as Lcam in Eq. (3), or only the latter? Please specify which camera tokens and losses remain, since Lfm already supervises the global stream containing the camera and the two interpretations test different hypotheses.
  2. [Clarification; Weakness 1] Do existing experiments compare joint and separate hand/camera prediction pathways while retaining the same conditioning and camera supervision? If so, please give the relevant result and configuration. If not, please clarify that Table 4(a) identifies the benefit of camera supervision within the joint model; the matched factorized comparison suggested above is a future validation direction, not a request for a new rebuttal training campaign.
  3. [Clarification; Weakness 2] For Table 3, are the alignment transforms for W-MPJPE, WA-MPJPE, and wrist-level WA-MPJPE fitted jointly to both hands or separately, and over which frames? Please also identify the valid-frame masks and aggregation used for these scores versus MRRPE-A. The existing protocol in Appendix B would help establish how the aligned trajectory gains relate to the inter-hand error increase; that appendix is not included in the supplied 12-page PDF.
  4. [Clarification; Weakness 2] Do the existing H2O/TACO predictions or analyses localize the MRRPE-A increase to missing-hand intervals, a particular hand, or another recurring error? Please also state how the five inference samples are combined or selected, whether this uses any ground truth, and whether existing individual-sample versus aggregate results show a relative-placement trade-off (Sec. 3.4 and referenced Appendix C.1). Existing analyses are sufficient for the rebuttal; any unperformed visibility-stratified or aggregation analysis can be identified as future work.
4 comments
AuthorsOpenReview ↗

Response to Reviewer du7s (1/2): Q1, Q2

We thank the reviewer for the careful analysis and concrete suggestions.

Q1. What Table 4(a) removes.

We remove both the camera term of L_fm (Eq. 2) and L_cam (Eq. 3). The camera tokens remain as an unsupervised noise stream. We will state this in Sec. 4.4.

Q2 / W1. Auxiliary supervision vs. joint prediction.

Thank you for suggesting this distinction. We examine attention coupling and joint prediction through two complementary comparisons.

Matched attention-coupling comparison. We trained factorized and coupled predictors with identical encoders, data, camera supervision, training recipes, and training schedules. Both use a one-way condition stream. Factorized additionally masks all hand–camera attention, preventing communication through either direct attention or the condition stream.

HOT3DWA ↓wWA ↓AccEr ↓
Factorized35.8729.161.944
Coupled35.9829.331.884

WA and wWA use per-hand SE(3) alignment (mm). AccEr is measured in m/s². Positional accuracy is comparable, while Coupled has 3.1% lower acceleration error. Thus, this matched comparison does not establish a positional benefit from direct hand–camera attention.

Joint vs. single-task prediction. Both predictors above retain both output tasks and their supervision. We therefore additionally trained hand-only and camera-only predictors, each predicting only its designated output while retaining the visual conditioning. Each entry below reports Full / hand-only, using the same hand metrics.

DatasetWA ↓wWA ↓AccEr ↓
HOT3D36.48 / 36.4629.70 / 30.131.892 / 2.008
HoloAssist52.90 / 53.3842.67 / 42.627.385 / 7.476
H2O19.53 / 19.6014.21 / 14.242.596 / 2.732
ARCTIC87.16 / 87.6771.64 / 71.712.931 / 3.154
TACO29.37 / 32.2922.16 / 24.123.348 / 3.506

Full reduces hand acceleration error across all five datasets by 1.2–7.1% and improves zero-shot world-space hand placement on TACO, reducing WA/wWA by 9.1%/8.1%. All four datasets other than HOT3D are evaluated zero-shot using HOT3D-trained models.

For cameras, each entry reports Full / camera-only. ATE SE(3) uses one rigid alignment per 150-frame window without scale adjustment. rot is the mean absolute rotation error, and RPE_t@1 is the RMSE of relative translation over one-frame intervals. Translation errors are in mm and rotation errors are in degrees.

DatasetATE SE(3) ↓rot ↓RPE_t@1 ↓
HOT3D16.82 / 16.891.80 / 1.653.56 / 5.62
HoloAssist33.23 / 29.955.21 / 5.434.33 / 5.99
H2O5.12 / 5.021.10 / 1.221.28 / 1.63
ARCTIC41.42 / 38.895.15 / 5.415.26 / 6.99
TACO10.94 / 11.191.90 / 1.932.35 / 3.06

Full reduces translation RPE_t@1 across all five datasets by 21.4–36.7%. Absolute rotation error also decreases on all four zero-shot datasets, while ATE SE(3) shows mixed results. These results suggest a benefit particularly in short-interval camera motion reconstruction.

Together, these results suggest benefits of the joint formulation, particularly in temporal reconstruction and zero-shot hand placement. Direct hand–camera attention yields comparable positional accuracy in the matched comparison. We will add these comparisons to Sec. 4.4 and clarify that Table 4(a) evaluates the removal of camera supervision.

AuthorsOpenReview ↗

Response to Reviewer du7s (2/2): Q3-Q4

Q3 / Q4(a) / W2. Alignment and relative-placement errors.

We summarize the alignment and valid-frame definitions from Appendix B of the supplementary PDF, included in the submitted ZIP, and clarify interpolation and score aggregation. Across the original tables, W / WA / wWA align each hand on its GT-annotated frames. W uses the first two such frames, and WA / wWA use all of them. Missing baseline frames are interpolated, and scores are averaged over hand-clips. MRRPE-A uses frames where both hands are GT-annotated, fits one shared rotation to both hands’ joints, applies it to the inter-wrist vector without scaling, and averages over windows.

With a shared two-hand SE(3) alignment, our original model also has the lowest WA-MPJPE among the evaluated methods on HOT3D / ARCTIC (44.78 / 41.64 mm). Below are two-hand WA / wWA on the transfer datasets (mm), fitting one transform to both hands using all joints / wrists, respectively. We use frames with GT for both hands and finite predictions from every method, averaging over 284 / 160 / 130 windows for HoloAssist / H2O / TACO.

MethodHoloAssistH2OTACO
HaWoR84.22 / 76.5340.65 / 35.8148.76 / 44.99
HandFlow169.15 / 163.2329.57 / 26.7146.53 / 42.68
ReViV129.47 / 119.4735.14 / 27.9553.34 / 47.69
WiLoR+ViPE185.38 / 182.8926.61 / 19.7138.04 / 35.78
EgoForce+ViPE198.98 / 193.4125.03 / 20.7949.25 / 46.39
HOPformer+ViPE168.81 / 152.3044.19 / 30.1765.12 / 49.95
Ours (zero-shot)63.54 / 54.7634.01 / 24.3339.44 / 34.37
Ours (additional training)59.91 / 51.2120.31 / 16.6825.49 / 23.77

ReViV was trained on all three target datasets, and EgoForce was trained on H2O. Our original zero-shot model leads in per-hand WA on all three transfer datasets (Table 3) and in shared two-hand WA on HoloAssist. Baselines achieve lower shared two-hand WA on H2O and TACO.

Following the reviewer’s suggestion, we stratified the existing predictions by hand visibility. Both hands are detected in 99.9% / 94.5% of the evaluated bimanual frames on H2O / TACO, and the excess relative-placement error is concentrated in these detected-hand frames. Elevated inter-hand errors occur during H2O spray/lotion actions and fine-grained tool manipulation with the right hand in TACO, despite low per-hand aligned wrist errors.

We believe limited coverage of these interactions in our training data contributes to this gap. Consistent with this interpretation, additional training on the HOT3D, H2O and TACO training splits, without changing the architecture or losses, reduces MRRPE-A from 46.17 to 30.89 mm on H2O and from 60.95 to 40.44 mm on TACO. The resulting model achieves the lowest two-hand WA / wWA among the evaluated methods on all three transfer datasets. HoloAssist remains zero-shot. We will include failure cases and videos in the supplementary material.

Q4(b). How the five inference samples are combined.

We use velocity averaging. At every Euler step, we average the five samples’ predicted endpoints, so the samples converge to a single output. No sample is selected and no GT is used. Velocity averaging reduces MRRPE-A on all five datasets.

InferenceHOT3DARCTICHoloAssistH2OTACO
Single sample60.7051.3889.2046.4662.33
Velocity average (ours)56.5149.5383.9845.9260.67
Reviewer_du7sOpenReview ↗

Remaining concerns about joint prediction and bimanual generalization

Thank you for your response. The added experiments and protocol details are helpful. My questions about the camera losses and sample aggregation have been answered, but the new results only partly resolve my broader concerns.

The matched factorized-versus-coupled comparison answers an important question. With the same conditioning and camera supervision, WA-MPJPE is 35.87 mm for the factorized model and 35.98 mm for the coupled model. This does not show a positional benefit from direct hand-camera attention. I appreciate the 3.1% reduction in acceleration error, and the separate single-task comparisons show useful gains in temporal accuracy and TACO hand placement. Still, hand-position errors are nearly unchanged on the other four datasets, and camera ATE is mixed. These results support a narrower benefit from joint prediction than the motivation about avoiding error propagation initially suggests.

The shared two-hand alignment results make the generalization concern more concrete. On H2O, zero-shot WA-MPJPE is 34.01 mm, compared with 26.61 mm for WiLoR+ViPE. On TACO, it is 39.44 versus 38.04 mm. HoloAssist remains a strong result, but the gains after aligning each hand separately do not consistently carry over to placing both hands correctly in one shared frame. For example, two individually accurate hand trajectories can still have the wrong distance or orientation relative to each other. That matters directly for the task this paper claims to solve.

Your visibility analysis also shows that these errors occur when both hands are detected, including spray/lotion actions and fine-grained tool manipulation. Adding H2O and TACO training data improves the results, but it changes the evaluation setting and does not resolve the original zero-shot limitation on those datasets.

I recognize the practical accuracy gains and the value of directly predicting metric world-space motion. However, the limited positional benefit from direct coupling leaves me less convinced about the size of the methodological advance over existing generative hand-motion and joint hand-camera approaches. Together with the remaining bimanual transfer gap, this leaves me at my original assessment.

AuthorsOpenReview ↗

Clarifying the scope of our contributions

Thank you for the careful follow-up and for engaging with our additional experiments. We had not sufficiently examined the effect of hand–camera attention coupling or the limits of zero-shot transfer, and the reviewer’s points were important. The additional experiments and this discussion have made the scope of our contributions clearer and strengthened the reliability of our findings. We will revise the paper accordingly.

We agree that the matched comparison does not show a positional benefit from direct hand–camera attention. Both the factorized and coupled variants generate hands directly in the world frame, so this comparison isolates the attention coupling, not the formulation itself. Compared with single-task prediction, joint prediction mainly improves temporal reconstruction: hand AccEr by 1.2–7.1% and camera RPE_t@1 by 21.4–36.7% across five datasets (Q2). We will revise Sec. 1 and Sec. 4.4 to state these benefits precisely, and to separate the design choice of avoiding explicit coordinate composition from its measured effect on accuracy.

We also agree that zero-shot bimanual placement on H2O and TACO remains a limitation. We will state this explicitly and report the additional-training results separately from the zero-shot results. With shared two-hand alignment, the original model is best on HOT3D and ARCTIC (44.78 / 41.64 mm) and on zero-shot HoloAssist (63.54 vs. 84.22 mm for HaWoR) (Q3).

With this scope clarified, we believe the central contribution remains: directly generating metric world-space motion of both hands and the camera, instead of composing separately estimated hands and camera as in prior pipelines. In a matched coordinate-frame ablation on HOT3D, we retrained the same architecture with only the hand coordinate frame changed. Predicting hands directly in the world frame gives 34.83 mm WA-MPJPE, versus 38.54 mm when camera-space hands are composed with our generated camera and 65.65 mm with ViPE (606 windows; all variants were retrained for this comparison, so the values differ slightly from Table 4). Overall, DewHand reduces WA-MPJPE relative to StableHand from 43.74 to 28.56 mm on the HOT3D StableHand split and from 78.63 to 36.35 mm on ARCTIC (Table 1, predicted shape). It recovers camera scale within 10% on 66.2% of clips, versus 24.6% for ViPE (Table 2). Flow matching improves over same-architecture regression in both WA-MPJPE (45.13 to 35.94 mm) and AccEr (3.34 to 1.89 m/s²) (Table 4(c)).

We hope these clarifications are helpful, and we would be grateful if the reviewer could take these contributions and results into account in the final assessment.

Reviewer_qxZQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper reconstructs metric world-space two-hand and camera motion from monocular egocentric video. Prior multi-stage methods estimate hands and camera separately, so camera drift and inconsistencies propagate to hand motion. DewHand is a single-stage generative model that (1) conditions on scene-level features from a frozen VGGT-Ω and hand-specific features from a frozen HaMeR; (2) uses a global stream and a local stream coupled through MMDiT blocks with alternating within-stream and cross-stream attention; and (3) uses flow matching instead of deterministic regression to handle occlusion and out-of-view ambiguity. Training combines the flow-matching loss, metric-space joint/camera losses, and wrist velocity/acceleration losses; inference uses a few Euler steps and multiple samples.

On HOT3D and ARCTIC, DewHand lowers rigid-aligned world-space error versus StableHand and recovers metric scale far more reliably than ViPE/DROID-SLAM-based methods. Zero-shot results on HoloAssist, H2O, and TACO give the lowest world-aligned error. Ablations isolate camera joint training, feature sources, and training objective.

Critical strengths
  • Clear evidence for joint generation. Removing camera supervision worsens SE(3)-aligned WA-MPJPE (35.94→42.36 mm) while PA-MPJPE barely changes.
  • Strong metric-scale recovery. 66.2% of clips within 10% scale error vs. HaWoR 26.1%, ViPE 24.6%, and even the frozen VGGT-Ω backbone itself (20.6%).
  • Large SOTA gains, e.g., ARCTIC SE(3)-WA-MPJPE 78.63→36.35 mm vs. StableHand (predicted shape), with advantages also under GT shape.
  • Occlusion robustness. With over half the frames lacking hand detections, error is 71.0 mm vs. 134.3 mm for the best baseline.
  • Cross-device zero-shot generalization from Aria to HoloLens2, H2O, and TACO, without intrinsics.
  • Well-designed feature ablation showing geometric features help placement and hand features help articulation.
Critical weaknesses
  • Possible test-time compute confound in the flow-matching claim. It is unclear whether the L2 regression baseline (Table 4c) also gets 5-sample aggregation; regression cannot produce diverse samples, so the gain (WA-SE(3) 45.13→35.94 mm) may partly reflect ensembling. The 1-step ablation does not resolve this.
  • Unexplained extreme loss weights. The velocity/acceleration weights are 800 vs. 1 for joint/camera losses, with no sensitivity analysis.
  • Fixed 150-frame offline window. Non-causal attention over a 5-second clip limits applicability to long videos; stitching strategies and boundary drift are not discussed, creating a gap with the embodied-agent motivation.
  • Zero-shot results are mixed. On H2O, PA-MPJPE (6.80) is worse than WiLoR+ViPE (5.68) and EgoForce+ViPE (6.08), and MRRPE-A is not best on H2O/TACO; the abstract's WA-MPJPE claim reads broader than the evidence.
  • Reliance on frozen VGGT-Ω. Failure modes (low texture, blur, fast head rotation) and possible pretraining-data overlap with evaluation sets are not analyzed.
Questions for authors
  1. Does the L2 regression ablation use the same 5-sample aggregation? Please compare under matched test-time compute (flow matching with 1 sample vs. regression with 1 pass).
  2. How were λv = λa = 800 chosen, and how sensitive are Tables 1–2 to these weights?
  3. How are clips longer than 150 frames handled (sliding window, overlap averaging)? Is there drift or discontinuity at seams, and can you show results on 1-minute or longer sequences?
  4. Why are PA-MPJPE and MRRPE-A not best on H2O/TACO: HaMeR feature generalization limits, or a trade-off from metric-space losses?
  5. Have you checked VGGT-Ω pretraining overlap with the evaluation datasets, and can you correlate VGGT-Ω camera accuracy with DewHand's final accuracy per sequence?
3 comments
AuthorsOpenReview ↗

Response to Reviewer qxZQ (1/2): W1-W5

We thank the reviewer for the detailed questions and constructive feedback.

W1 / Q1. Flow matching under matched inference compute.

Following the reviewer’s suggestion, we conducted a matched-compute comparison. The L2-regression row of Table 4(c) was evaluated with the same inference as ours (2 steps × 5 samples, 10 head calls), and it does not benefit from it: a single call gives the same 45.15 mm. Under matched compute (one head call each, HOT3D, WA-MPJPE SE(3)):

MethodHead callsWA-MPJPE SE(3) ↓
Regression145.15
Flow, 1 step, 1 sample140.44
Flow, paper inference1035.99

At identical compute, flow matching is 4.7 mm better, so the gain persists under matched one-call compute.

W2 / Q2. Velocity and acceleration weights.

We chose λᵥ = λₐ = 800 to bring the smaller raw temporal losses to roughly the same scale as the other losses. A HOT3D sensitivity sweep gives:

λᵥ = λₐPA-MPJPE ↓WA-MPJPE SE(3) ↓AccEr ↓Camera ATE SE(3) / rot. ↓
2006.8239.112.4016.61 / 1.73°
800 (paper)6.7535.991.8916.77 / 1.73°
32006.7733.671.6416.61 / 1.78°

Camera ATE SE(3) uses one rigid alignment per 150-frame window without scale adjustment; translation errors are in mm and rotation errors are in degrees.

The temporal weights affect trajectory accuracy and acceleration error, while PA-MPJPE and camera accuracy remain largely stable. We will include this sweep in the appendix.

W3 / Q3. Long videos and online use.

We process videos in 150-frame windows with a 50-frame overlap and blend predictions in the overlap at each Euler step. Averaging reduces wrist displacement at window boundaries from 65.4 to 6.5 mm/frame (GT: 7.0), with unchanged per-window WA-MPJPE (35.96 vs. 35.99 mm). On 27 full HOT3D sequences (43–149 s), DewHand achieves the lowest W- and WA-MPJPE, with camera ATE on par with StableHand (mm):

MethodW-MPJPE SE(3) ↓WA-MPJPE SE(3) ↓Camera ATE SE(3) ↓
HaWoR501.8268.5270.9
HandFlow365.9242.3222.4
ReViV*548.7270.9257.9
EgoForce+ViPE442.7237.4222.4
StableHand*238.2118.079.7
Ours206.5109.779.6

* Overlapping windows linked by SE(3) fits to their own cameras on the overlap. StableHand uses GT hand shape.

Window boundaries are smooth, but camera drift remains: on the nine of the 27 sequences that reach 120 s, median camera-position error after a single SE(3) alignment on the first window increases from 28.3 mm at 10 s to 179.5 mm at 120 s. The windows are linked without loop closure; global optimization across windows, as in SLAM back-ends, is a possible extension. Online inference is also future work. One possible approach is to combine a streaming 3D backbone with causal attention in the hand–camera head, using a few anchor frames and a sliding window of recent frames.

W4 / Q4. PA-MPJPE and MRRPE-A on H2O / TACO.

HaMeR features may contribute to the PA-MPJPE gap: on H2O, frozen HaMeR itself has higher PA-MPJPE than WiLoR+ViPE (6.34 vs. 5.68 mm). Our analysis finds elevated MRRPE-A during hand–hand interactions on H2O (e.g., applying spray/lotion) and fine-grained tool manipulation on TACO. We believe the gap partly reflects their rarity in our original training data. Additional training on HOT3D/H2O/TACO, with unchanged architecture and losses, reduces MRRPE-A (mm):

MethodH2OTACO
Best baseline22.76 (EgoForce+ViPE)54.28 (ReViV)
Ours (original)46.1760.95
Ours (+ additional training)30.8940.44

After additional training on the HOT3D/H2O/TACO training splits, the model achieves the lowest two-hand SE(3) WA-MPJPE among the evaluated methods on HoloAssist/H2O/TACO (59.91/20.31/25.49 mm). HoloAssist remains zero-shot.

W5 / Q5. VGGT-Ω failure modes and correlation.

Fast head rotation has the strongest association with DewHand’s WA-MPJPE SE(3) on HOT3D (Spearman ρ = 0.51 across 606 windows); blur and low texture have weaker associations. Even in the hardest third, we beat HaWoR on 89–93% of windows.

Across 27 sequences, mean DewHand WA-MPJPE SE(3) correlates with mean VGGT-Ω camera ATE (Spearman ρ = 0.60). HaWoR, which does not use VGGT-Ω, also degrades on the same windows (2.9× vs. our 2.3× from the easiest to the hardest third). Our generated camera more than halves VGGT-Ω’s ATE in every third.

AuthorsOpenReview ↗

Response to Reviewer qxZQ (2/2): Q5

W6 / Q5. Pretraining overlap.

Thank you for raising this. HaMeR is not trained on our evaluation sets, but VGGT-Ω’s training data includes HOT3D. Our backbone ablation (Table 4(b)) includes Depth Anything 3 (DA3), whose training data contains none of our evaluation datasets. A common-set re-evaluation gives:

BackboneWA-MPJPE SE(3) ↓PA-MPJPE ↓Camera ATE SE(3) ↓
VGGT-Ω35.996.7516.77
DA336.396.7717.37

All errors are in mm. Camera ATE uses one rigid alignment per 150-frame window without scale adjustment.

These results show that comparable HOT3D performance is achievable without HOT3D overlap in backbone pretraining. HoloAssist, H2O and TACO are in neither backbone’s training data. We will explicitly disclose VGGT-Ω’s HOT3D overlap and include this comparison in the revised paper.

Thank you to the authors for the detailed rebuttal and the substantial additional experiments. I appreciate the effort taken to address my concerns.

Regarding the zero-shot results on H2O and TACO, the additional analysis provides a plausible explanation for the performance gap, particularly the limitations of frozen HaMeR features under hand-hand interactions and fine-grained tool manipulation. However, I believe the original zero-shot results and the results obtained after additional dataset-specific training should be clearly distinguished in the revised paper.

My main remaining concern is the gap between the paper's embodied-agent motivation and its current inference paradigm. Although the proposed window-stitching strategy enables long-video reconstruction, the model still relies on non-causal processing of 150 frame windows, and the authors explicitly acknowledge that online inference remains future work. In addition, the reported camera-position error increase to 179.5 mm after 120 seconds indicates that long-term global trajectory consistency remains challenging.

Overall, the rebuttal substantially addresses most of my technical concerns and strengthens the experimental evidence supporting the proposed method. I appreciate the authors' thorough responses and believe the remaining limitations can be appropriately discussed in the revised paper.

Reviewer_bH9xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DewHand addresses the recovery of two-hand motion in a metric world frame from monocular egocentric video. It treats the camera trajectory and both hands as one joint generative target. A flow-matching model with an MMDiT-style backbone operates over three token streams: a global-motion stream carrying camera and per-hand global translation/rotation, a local hand stream carrying MANO pose and shape, and a condition stream. Training combines the flow-matching objective with auxiliary joint, camera, velocity and acceleration losses. Inference runs on 150-frame windows with only two Euler steps and five samples. Validation on HOT3D and ARCTIC uses 150-frame, 5-second clips. DewHand reports large gains over compared methods.

Critical strengths

(a) Strong metric-scale and camera results, corroborated across protocols. Metric scale is the core difficulty in monocular egocentric reconstruction, and DewHand recovers scale within 10% on two thirds of clips (vs. roughly a quarter for SLAM), with small gaps between Sim(3)- and SE(3)-aligned hand errors and large camera ATE improvements. Cross-device zero-shot transfer (Aria fisheye to HoloLens perspective, plus H2O and TACO) with the lowest rigid-aligned errors is meaningful evidence of robustness rather than in-domain fitting.

(b) The ablations isolate the two claims that actually carry the paper. Removing camera supervision degrades hand error from 28.56 to 42.36 — i.e. most of the gain over the decomposed baseline disappears — which substantiates the joint-modelling claim rather than leaving it to be inferred from the end-to-end number. The L2-regression variant (45.13) separately shows the generative formulation is load-bearing and not decorative. Combined with two-step inference, this is a coherent, cheap, reproducible-in-principle design.

Critical weaknesses

The model is restricted to fixed 5-second offline windows and ignores manipulated objects and contact; ARCTIC results are fine-tuned (in-domain), and on the zero-shot datasets only the SE(3)-aligned metrics lead while local-pose and inter-hand metrics are mixed. The practical value of the generative multi-sample design is not demonstrated (diversity increases precisely where error is high, but best-of-N or aggregated samples are not shown to help a downstream user; five samples cost roughly fivefold).

Moreover, "single-stage" inference still requires running two heavy frozen backbones (VGGT-Omega and HaMeR) over every clip, so the speed advantage over SLAM may be overstated if that cost is excluded.

Questions for authors

(1) Does the reported runtime include the frozen VGGT-Omega and HaMeR forward passes over 150 frames? Please give end-to-end latency on identical hardware versus the SLAM pipelines.

(2) On H2O and TACO, DewHand is behind on PA-MPJPE and MRRPE-A. Is the local hand stream ever better than frozen HaMeR on data it has not seen? If not, would a variant that predicts only camera/global motion and passes HaMeR's local pose through unchanged match or beat the full model?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bH9x (1/2): W1–W4

We thank the reviewer for the constructive feedback and suggestions.

W1. Fixed 5-second offline windows; objects and contact.

We process videos in 150-frame windows with a 50-frame overlap and average predictions in the overlap at each Euler step. Averaging reduces wrist displacement at window boundaries from 65.4 to 6.5 mm/frame (GT: 7.0), with unchanged per-window WA-MPJPE (35.96 vs. 35.99 mm). On 27 full HOT3D sequences (43–149 s), DewHand achieves the lowest W- and WA-MPJPE, with camera ATE on par with StableHand (mm):

MethodW-MPJPE SE(3) ↓WA-MPJPE SE(3) ↓Camera ATE SE(3) ↓
HaWoR501.8268.5270.9
HandFlow365.9242.3222.4
ReViV*548.7270.9257.9
EgoForce+ViPE442.7237.4222.4
StableHand*238.2118.079.7
Ours206.5109.779.6

* Overlapping windows linked by SE(3) fits to their own cameras on the overlap. StableHand uses GT hand shape.

We see online hand tracking for embodied agents as a promising direction. One possible route is to combine a streaming 3D backbone with past-only attention and conditioning in the hand–camera head. We leave this extension, along with modeling manipulated objects and contact, to future work.

W2. ARCTIC fine-tuning and mixed zero-shot results.

The ARCTIC comparison is in-domain for the baselines as well: WiLoR+ViPE, HaWoR, EgoForce+ViPE, HOPformer+ViPE and StableHand are all trained on ARCTIC (only HandFlow, whose training code is not released, is zero-shot; Sec. 4.1, Table 1). We therefore do not use ARCTIC to support generalization; that claim rests on HoloAssist, H2O and TACO. We lead on every metric on HoloAssist and on AccEr across all three datasets, while acknowledging the gaps on H2O and TACO, particularly PA-MPJPE and MRRPE-A, in the conclusion. Our zero-shot claim specifically refers to SE(3)-aligned WA-MPJPE under the per-hand protocol (Table 3), which is lowest on all three datasets. We will revise the abstract to state this metric and protocol explicitly.

W3. Cost and benefit of velocity averaging.

VGGT-Ω and HaMeR features are computed once and shared across the five samples; only the flow head is evaluated per sample. Five-sample velocity averaging takes about 1.2× the runtime of single-sample inference. We will clarify this feature reuse and runtime overhead in Appendix C. Velocity averaging improves accuracy on every dataset (WA-MPJPE SE(3)):

InferenceHOT3DARCTICHoloAssistH2OTACO
Single sample37.9637.2255.1521.3432.51
Velocity average (ours)35.7736.3952.5320.6431.23

W4 / Q1. End-to-end inference cost.

As reported in the supplementary material, our end-to-end timings already include both frozen backbones. On a single A100, DewHand processes a 150-frame clip in 12.0 s end to end, compared with 31.4–44.9 s for HandFlow, WiLoR+ViPE, and HaWoR. These timings include each method’s preprocessing, including detection, feature extraction, and SLAM where applicable. For DewHand, VGGT-Ω and the flow head account for 1.7 s in total, with VGGT-Ω processing the full clip at 256 px. The remaining time goes to hand detection and HaMeR feature extraction.

AuthorsOpenReview ↗

Response to Reviewer bH9x (2/2): Q2

Q2. Local hand stream vs. frozen HaMeR, and passing HaMeR’s pose through.

How does the local stream compare with frozen HaMeR on unseen datasets?

On frames where HaMeR detects the hand, DewHand achieves lower PA-MPJPE on HoloAssist, nearly identical error on TACO, and higher error on H2O. Each entry reports DewHand / Frozen HaMeR.

DatasetPA-MPJPE (mm) ↓
HoloAssist11.74 / 12.99
H2O6.80 / 6.34
TACO9.20 / 9.24

Does using HaMeR’s finger poses match the full model?

We evaluated the suggested variant by retaining DewHand’s wrist positions, global orientations, camera predictions and hand shape, and replacing only the finger poses with HaMeR’s. Each entry reports Full / HaMeR-finger variant.

DatasetPA-MPJPE ↓WA-MPJPE SE(3) ↓AccEr ↓
HoloAssist11.96 / 13.2252.67 / 52.987.33 / 8.72
H2O6.80 / 6.3420.68 / 20.932.60 / 3.38
TACO9.26 / 9.2931.29 / 31.283.35 / 4.69

Both variants use the same evaluation frames and global-motion predictions. Unlike the detection-only comparison above, this table includes all valid evaluation frames, with missing HaMeR poses filled using the existing interpolation procedure. Position errors are in mm; WA uses per-hand SE(3) alignment, and AccEr is in m/s².

HaMeR fingers improve local-pose accuracy on H2O, while world-aligned accuracy remains comparable or slightly worse. However, acceleration error increases by 1.2–1.4× across these three datasets. MRRPE-A changes by at most 0.25 mm, indicating that substituting finger poses does not resolve the relative-placement gap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.