- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DewHand jointly generates metric world-space hand and camera trajectories from monocular egocentric video using geometric and hand-specific features with flow matching (Secs. 3.1–3.4). Experiments report improved rigid-aligned hand reconstruction on HOT3D and ARCTIC, with additional camera evaluation, cross-device transfer, and component ablations (Tables 1–4).
Critical strengths
- With predicted hand shape, DewHand reduces SE(3)-aligned WA-MPJPE relative to StableHand from 43.74 to 28.56 mm on the HOT3D StableHand split and from 78.63 to 36.35 mm on ARCTIC (Table 1), providing substantial evidence for improved metric world-space reconstruction without privileged shape input.
- The controlled ablations support the design choices: removing camera supervision raises WA-MPJPE from 35.94 to 42.36 mm, while replacing flow matching with same-architecture regression raises it to 45.13 mm (Table 4; lines 430–468).
Critical weaknesses
- The experiments establish the benefit of camera supervision, but do not isolate the contribution of coupling hand and camera prediction to the proposed solution to error propagation. Joint generation is a central motivation (p. 2, lines 80–103), and the architecture exchanges information among hand and camera tokens (p. 5, lines 231–237). Table 4(a), however, changes camera supervision: WA-MPJPE improves from 42.36 to 35.94 mm. This supports the authors' stated supervision benefit, but does not distinguish useful auxiliary supervision from a benefit of joint prediction over separate prediction pathways supplied with the same geometric features and supervision. The comparisons to differently constructed multi-stage systems in Table 1 do not resolve that distinction. Consequently, the complete system's accuracy gains are convincing, while the architectural explanation for those gains remains less secure. A matched factorized hand/camera predictor retaining the encoders, training data, and camera supervision would be a focused future test; any already available control or a narrower interpretation of the current ablation would address the immediate concern.
- The transfer results expose a substantial weakness in relative two-hand geometry despite improved rigid-aligned trajectory accuracy. On H2O, DewHand improves SE(3)-aligned WA-MPJPE over WiLoR + ViPE from 22.04 to 20.68 mm, but nearly doubles MRRPE-A from 23.76 to 46.17 mm; its Sim(3)-aligned WA-MPJPE is also worse (19.54 versus 14.80 mm). On TACO, MRRPE-A is 60.95 versus 55.82 mm despite better SE(3)-aligned WA-MPJPE (Table 3, p. 8). MRRPE-A measures inter-wrist displacement after a shared rigid alignment (p. 6, lines 311–314), making this a limitation of the stated common-world-frame reconstruction task, rather than an unrelated downstream requirement. The authors acknowledge mixed transfer results (lines 423–425), and HoloAssist improves across metrics, so this is not evidence of universal transfer failure. It does show that improved scale-preserving trajectory scores alone do not establish reliable bimanual spatial relationships across devices. Examining existing predictions by hand visibility and relative-wrist error, with explicit alignment and frame-selection definitions, would help determine the failure regime and the appropriate scope of the generalization claim.
Questions for authors
- [Clarification; Weakness 1] In the existing Table 4(a) implementation, does “w/o camera supervision” remove the camera components from Lfm in Eq. (2) as well as Lcam in Eq. (3), or only the latter? Please specify which camera tokens and losses remain, since Lfm already supervises the global stream containing the camera and the two interpretations test different hypotheses.
- [Clarification; Weakness 1] Do existing experiments compare joint and separate hand/camera prediction pathways while retaining the same conditioning and camera supervision? If so, please give the relevant result and configuration. If not, please clarify that Table 4(a) identifies the benefit of camera supervision within the joint model; the matched factorized comparison suggested above is a future validation direction, not a request for a new rebuttal training campaign.
- [Clarification; Weakness 2] For Table 3, are the alignment transforms for W-MPJPE, WA-MPJPE, and wrist-level WA-MPJPE fitted jointly to both hands or separately, and over which frames? Please also identify the valid-frame masks and aggregation used for these scores versus MRRPE-A. The existing protocol in Appendix B would help establish how the aligned trajectory gains relate to the inter-hand error increase; that appendix is not included in the supplied 12-page PDF.
- [Clarification; Weakness 2] Do the existing H2O/TACO predictions or analyses localize the MRRPE-A increase to missing-hand intervals, a particular hand, or another recurring error? Please also state how the five inference samples are combined or selected, whether this uses any ground truth, and whether existing individual-sample versus aggregate results show a relative-placement trade-off (Sec. 3.4 and referenced Appendix C.1). Existing analyses are sufficient for the rebuttal; any unperformed visibility-stratified or aggregation analysis can be identified as future work.
4 comments
Response to Reviewer du7s (2/2): Q3-Q4
Q3 / Q4(a) / W2. Alignment and relative-placement errors.
We summarize the alignment and valid-frame definitions from Appendix B of the supplementary PDF, included in the submitted ZIP, and clarify interpolation and score aggregation. Across the original tables, W / WA / wWA align each hand on its GT-annotated frames. W uses the first two such frames, and WA / wWA use all of them. Missing baseline frames are interpolated, and scores are averaged over hand-clips. MRRPE-A uses frames where both hands are GT-annotated, fits one shared rotation to both hands’ joints, applies it to the inter-wrist vector without scaling, and averages over windows.
With a shared two-hand SE(3) alignment, our original model also has the lowest WA-MPJPE among the evaluated methods on HOT3D / ARCTIC (44.78 / 41.64 mm). Below are two-hand WA / wWA on the transfer datasets (mm), fitting one transform to both hands using all joints / wrists, respectively. We use frames with GT for both hands and finite predictions from every method, averaging over 284 / 160 / 130 windows for HoloAssist / H2O / TACO.
| Method | HoloAssist | H2O | TACO |
|---|---|---|---|
| HaWoR | 84.22 / 76.53 | 40.65 / 35.81 | 48.76 / 44.99 |
| HandFlow | 169.15 / 163.23 | 29.57 / 26.71 | 46.53 / 42.68 |
| ReViV | 129.47 / 119.47 | 35.14 / 27.95 | 53.34 / 47.69 |
| WiLoR+ViPE | 185.38 / 182.89 | 26.61 / 19.71 | 38.04 / 35.78 |
| EgoForce+ViPE | 198.98 / 193.41 | 25.03 / 20.79 | 49.25 / 46.39 |
| HOPformer+ViPE | 168.81 / 152.30 | 44.19 / 30.17 | 65.12 / 49.95 |
| Ours (zero-shot) | 63.54 / 54.76 | 34.01 / 24.33 | 39.44 / 34.37 |
| Ours (additional training) | 59.91 / 51.21 | 20.31 / 16.68 | 25.49 / 23.77 |
ReViV was trained on all three target datasets, and EgoForce was trained on H2O. Our original zero-shot model leads in per-hand WA on all three transfer datasets (Table 3) and in shared two-hand WA on HoloAssist. Baselines achieve lower shared two-hand WA on H2O and TACO.
Following the reviewer’s suggestion, we stratified the existing predictions by hand visibility. Both hands are detected in 99.9% / 94.5% of the evaluated bimanual frames on H2O / TACO, and the excess relative-placement error is concentrated in these detected-hand frames. Elevated inter-hand errors occur during H2O spray/lotion actions and fine-grained tool manipulation with the right hand in TACO, despite low per-hand aligned wrist errors.
We believe limited coverage of these interactions in our training data contributes to this gap. Consistent with this interpretation, additional training on the HOT3D, H2O and TACO training splits, without changing the architecture or losses, reduces MRRPE-A from 46.17 to 30.89 mm on H2O and from 60.95 to 40.44 mm on TACO. The resulting model achieves the lowest two-hand WA / wWA among the evaluated methods on all three transfer datasets. HoloAssist remains zero-shot. We will include failure cases and videos in the supplementary material.
Q4(b). How the five inference samples are combined.
We use velocity averaging. At every Euler step, we average the five samples’ predicted endpoints, so the samples converge to a single output. No sample is selected and no GT is used. Velocity averaging reduces MRRPE-A on all five datasets.
| Inference | HOT3D | ARCTIC | HoloAssist | H2O | TACO |
|---|---|---|---|---|---|
| Single sample | 60.70 | 51.38 | 89.20 | 46.46 | 62.33 |
| Velocity average (ours) | 56.51 | 49.53 | 83.98 | 45.92 | 60.67 |
Remaining concerns about joint prediction and bimanual generalization
Thank you for your response. The added experiments and protocol details are helpful. My questions about the camera losses and sample aggregation have been answered, but the new results only partly resolve my broader concerns.
The matched factorized-versus-coupled comparison answers an important question. With the same conditioning and camera supervision, WA-MPJPE is 35.87 mm for the factorized model and 35.98 mm for the coupled model. This does not show a positional benefit from direct hand-camera attention. I appreciate the 3.1% reduction in acceleration error, and the separate single-task comparisons show useful gains in temporal accuracy and TACO hand placement. Still, hand-position errors are nearly unchanged on the other four datasets, and camera ATE is mixed. These results support a narrower benefit from joint prediction than the motivation about avoiding error propagation initially suggests.
The shared two-hand alignment results make the generalization concern more concrete. On H2O, zero-shot WA-MPJPE is 34.01 mm, compared with 26.61 mm for WiLoR+ViPE. On TACO, it is 39.44 versus 38.04 mm. HoloAssist remains a strong result, but the gains after aligning each hand separately do not consistently carry over to placing both hands correctly in one shared frame. For example, two individually accurate hand trajectories can still have the wrong distance or orientation relative to each other. That matters directly for the task this paper claims to solve.
Your visibility analysis also shows that these errors occur when both hands are detected, including spray/lotion actions and fine-grained tool manipulation. Adding H2O and TACO training data improves the results, but it changes the evaluation setting and does not resolve the original zero-shot limitation on those datasets.
I recognize the practical accuracy gains and the value of directly predicting metric world-space motion. However, the limited positional benefit from direct coupling leaves me less convinced about the size of the methodological advance over existing generative hand-motion and joint hand-camera approaches. Together with the remaining bimanual transfer gap, this leaves me at my original assessment.
Clarifying the scope of our contributions
Thank you for the careful follow-up and for engaging with our additional experiments. We had not sufficiently examined the effect of hand–camera attention coupling or the limits of zero-shot transfer, and the reviewer’s points were important. The additional experiments and this discussion have made the scope of our contributions clearer and strengthened the reliability of our findings. We will revise the paper accordingly.
We agree that the matched comparison does not show a positional benefit from direct hand–camera attention. Both the factorized and coupled variants generate hands directly in the world frame, so this comparison isolates the attention coupling, not the formulation itself. Compared with single-task prediction, joint prediction mainly improves temporal reconstruction: hand AccEr by 1.2–7.1% and camera RPE_t@1 by 21.4–36.7% across five datasets (Q2). We will revise Sec. 1 and Sec. 4.4 to state these benefits precisely, and to separate the design choice of avoiding explicit coordinate composition from its measured effect on accuracy.
We also agree that zero-shot bimanual placement on H2O and TACO remains a limitation. We will state this explicitly and report the additional-training results separately from the zero-shot results. With shared two-hand alignment, the original model is best on HOT3D and ARCTIC (44.78 / 41.64 mm) and on zero-shot HoloAssist (63.54 vs. 84.22 mm for HaWoR) (Q3).
With this scope clarified, we believe the central contribution remains: directly generating metric world-space motion of both hands and the camera, instead of composing separately estimated hands and camera as in prior pipelines. In a matched coordinate-frame ablation on HOT3D, we retrained the same architecture with only the hand coordinate frame changed. Predicting hands directly in the world frame gives 34.83 mm WA-MPJPE, versus 38.54 mm when camera-space hands are composed with our generated camera and 65.65 mm with ViPE (606 windows; all variants were retrained for this comparison, so the values differ slightly from Table 4). Overall, DewHand reduces WA-MPJPE relative to StableHand from 43.74 to 28.56 mm on the HOT3D StableHand split and from 78.63 to 36.35 mm on ARCTIC (Table 1, predicted shape). It recovers camera scale within 10% on 66.2% of clips, versus 24.6% for ViPE (Table 2). Flow matching improves over same-architecture regression in both WA-MPJPE (45.13 to 35.94 mm) and AccEr (3.34 to 1.89 m/s²) (Table 4(c)).
We hope these clarifications are helpful, and we would be grateful if the reviewer could take these contributions and results into account in the final assessment.
Response to Reviewer du7s (1/2): Q1, Q2
We thank the reviewer for the careful analysis and concrete suggestions.
Q1. What Table 4(a) removes.
We remove both the camera term of
L_fm(Eq. 2) andL_cam(Eq. 3). The camera tokens remain as an unsupervised noise stream. We will state this in Sec. 4.4.Q2 / W1. Auxiliary supervision vs. joint prediction.
Thank you for suggesting this distinction. We examine attention coupling and joint prediction through two complementary comparisons.
Matched attention-coupling comparison. We trained factorized and coupled predictors with identical encoders, data, camera supervision, training recipes, and training schedules. Both use a one-way condition stream. Factorized additionally masks all hand–camera attention, preventing communication through either direct attention or the condition stream.
WA and wWA use per-hand SE(3) alignment (mm). AccEr is measured in m/s². Positional accuracy is comparable, while Coupled has 3.1% lower acceleration error. Thus, this matched comparison does not establish a positional benefit from direct hand–camera attention.
Joint vs. single-task prediction. Both predictors above retain both output tasks and their supervision. We therefore additionally trained hand-only and camera-only predictors, each predicting only its designated output while retaining the visual conditioning. Each entry below reports Full / hand-only, using the same hand metrics.
Full reduces hand acceleration error across all five datasets by 1.2–7.1% and improves zero-shot world-space hand placement on TACO, reducing WA/wWA by 9.1%/8.1%. All four datasets other than HOT3D are evaluated zero-shot using HOT3D-trained models.
For cameras, each entry reports Full / camera-only. ATE SE(3) uses one rigid alignment per 150-frame window without scale adjustment.
rotis the mean absolute rotation error, andRPE_t@1is the RMSE of relative translation over one-frame intervals. Translation errors are in mm and rotation errors are in degrees.Full reduces translation
RPE_t@1across all five datasets by 21.4–36.7%. Absolute rotation error also decreases on all four zero-shot datasets, while ATE SE(3) shows mixed results. These results suggest a benefit particularly in short-interval camera motion reconstruction.Together, these results suggest benefits of the joint formulation, particularly in temporal reconstruction and zero-shot hand placement. Direct hand–camera attention yields comparable positional accuracy in the matched comparison. We will add these comparisons to Sec. 4.4 and clarify that Table 4(a) evaluates the removal of camera supervision.