- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EgoFound3R combines a frozen geometry backbone, structured hand prompts, and multirate aggregation to predict world-space hands and point-wise visibility, contact, and distance from monocular egocentric video. Experiments report improved raw joint accuracy on three benchmarks and approximately sixfold throughput over WiLoR (Secs. 3–4; Tables 1–4 and 6).
Critical strengths
- Jointly decoding geometry and interaction attributes without an external hand detector or object template is a useful contribution for egocentric annotation, and the model improves both raw joint error and joint-level contact F1 on the three common evaluation sets (Secs. 3.2 and 3.4; Tables 1–2).
- The ablations substantiate the architectural choices: on TACO, replacing explicit points with MANO regression or removing hand-to-scene feedback increases raw joint error from 54.64 mm to 86.66 or 78.82 mm, respectively (Table 4). The shared-parameter stride study also directly examines the accuracy–efficiency trade-off (Appendices C and I, Fig. 10 and Tables 17–19).
Critical weaknesses
- The evidence for consistent world-space trajectories is weaker than the local hand-reconstruction results. World-space hands still compose camera-space predictions with a predicted camera trajectory, while scale refinement and overlapping-clip alignment remain post-processing (Appendix A, p. 15, lines 780–797). Sharing one scale within a clip does not by itself prevent camera-pose drift across clips. Yet all evaluation windows last 2 s, and the duration argument in Sec. 4.2, lines 318–323, is inferred from differences between datasets rather than sequence lengths. W and WA additionally fit a per-window Sim(3) using the first two or all valid frames, respectively (Appendix D, p. 18, lines 932–939 and 962–970), so these metrics do not independently validate absolute metric scale. On matched HOI4D windows, EgoFound3R's W error is worse than Dyn-HaMR's (205.48 versus 173.62 mm; Table 15). Table 10's temporal improvements and Table 20's TACO depth evaluation at the model's own scale are meaningful counterevidence, but neither establishes stability of stitched long videos. The claims should distinguish demonstrated short-window accuracy from long-video drift resistance; duration-controlled stitching evaluation would strengthen the latter in future work.
- The interaction results expose a material gap between accurate native-point predictions and dependable dense surface annotations. On TACO, contact F1 falls from 0.661 at 195 vertices to 0.320 at 778 vertices, with recall falling from 0.719 to 0.265 (Table 11); the matched dense comparison also favors InteractVLM, 0.458 versus 0.388 (Table 16). Moreover, distance MAE on ground-truth-contact native vertices is 31.38 mm (Table 19), exceeding the 14 mm band defining those contacts (Appendix G, p. 20, lines 1074–1079). This raises a substantive concern about using the outputs as contact/proximity supervision, beyond the missing explanation of attribute upsampling. The heads share features, but the described classification and distance objectives do not explain how their outputs are constrained to agree (Appendix B, Eqs. (4)–(8)); these aggregate errors do not themselves prove point-wise inconsistency. I acknowledge that Tables 13–14 preserve the native-point advantage under threshold and contact-definition changes, and Tables 17–19 support the architecture. Those results do not resolve dense-output degradation or establish precise proximity estimates. The paper should explain the usable resolution and consistency of its returned attributes and qualify its annotation claims accordingly.
Questions for authors
- [Clarification; Weakness 1] Does any existing result evaluate overlapping-clip stitching beyond the 2 s protocol? Please identify it and explain which evidence separates sequence duration from dataset difficulty in lines 318–323. If none does, can the drift claim be limited to the evaluated setting? A new long-video benchmark is not requested for rebuttal.
- [Clarification; Weakness 1] Please give the existing root-recovery, hand–depth scale-fusion, and clip-stitching rules, including the fallback when visible-hand support is weak (Appendix A, lines 774–797). Are Tables 1, 15, and 20 generated after the same scale-refinement procedure, and what does their combination establish about scale accuracy versus camera-trajectory error?
- [Clarification; Weakness 2] How are contact scores and ground-truth labels transferred from 195 to 778 vertices, and which threshold is used at each resolution? Can the existing implementation or cached evaluations explain the large recall loss and whether it arises at native vertices or newly interpolated vertices? Tables 13–14 currently address only the native point set.
- [Clarification; Weakness 2] What do existing outputs show about agreement between predicted contact and predicted distance, especially given TACO's 31.38 mm MAE on ground-truth contacts? Please clarify the evaluated distance quantity, any consistency mechanism beyond the losses shown, and the actual loss weights and stride-sampling proportions used for these results and ablations. Existing configurations or cached diagnostics would suffice; retraining is not requested.
3 comments
Thank you for your follow-up. We appreciate your recognition of the corrected interaction-attribute results and your clarification of the remaining concern regarding the world-space trajectories.
To examine the effect of scale alignment, we recomputed W and WA using rigid SE(3) alignment. SE(3) fits rotation and translation while fixing the scale to 1. The Sim(3) results below correspond to the corresponding rows of the revised Table 1. For each method, the Sim(3) and SE(3) results use the same valid samples.
| Dataset | Method | W Sim(3) | W SE(3) | WA Sim(3) | WA SE(3) |
|---|---|---|---|---|---|
| OakInk-v2 | HaWoR | 100.46 | 131.72 | 37.47 | 78.51 |
| ReViV | 80.66 | 82.49 | 29.45 | 49.28 | |
| EgoFound3R | 30.88 | 31.19 | 18.77 | 20.23 | |
| TACO | HaWoR | 86.38 | 99.51 | 35.34 | 59.86 |
| ReViV | 55.40 | 56.40 | 30.37 | 33.68 | |
| EgoFound3R | 40.03 | 40.97 | 25.16 | 27.38 | |
| HOI4D | HaWoR | 171.09 | 197.62 | 55.96 | 81.22 |
| ReViV | 260.61 | 292.29 | 62.37 | 136.12 | |
| EgoFound3R | 159.70 | 167.32 | 49.84 | 61.15 |
All errors are in mm, and lower is better.
We further evaluated stitched 120-frame long trajectories, each formed by concatenating adjacent 60-frame windows from the same sequence. The three methods are compared on their common valid hand frames.
| Dataset | Method | W Sim(3) | W SE(3) | WA Sim(3) | WA SE(3) |
|---|---|---|---|---|---|
| OakInk-v2 | HaWoR | 91.87 | 97.99 | 37.93 | 66.86 |
| ReViV | 175.89 | 163.74 | 43.70 | 92.86 | |
| EgoFound3R | 47.41 | 47.63 | 23.77 | 28.80 | |
| TACO | HaWoR | 171.47 | 199.07 | 43.97 | 109.52 |
| ReViV | 74.35 | 75.42 | 38.88 | 42.60 | |
| EgoFound3R | 44.74 | 45.02 | 28.37 | 30.70 | |
| HOI4D | HaWoR | 235.07 | 262.92 | 62.02 | 96.18 |
| ReViV | 376.93 | 414.49 | 69.69 | 183.14 | |
| EgoFound3R | 215.12 | 227.29 | 54.86 | 68.93 |
The SE(3) columns in both tables fix the scale while still fitting rotation and translation, so they can be compared directly with the Sim(3) results on the same samples. The close agreement between the Sim(3) and SE(3) errors indicates that EgoFound3R's predicted trajectories are already metrically accurate, so removing per-window scale fitting changes the W/WA errors only marginally.
We hope these additional results help clarify the metric-scale behavior of the composed world-space hand trajectories. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.
We would also like to provide some context for the trajectory lengths used in our evaluation. ReViV evaluates hand reconstruction on 60-frame clips, Dyn-HaMR evaluates global hand motion on 128-frame segments, and HaWoR's released evaluation code computes W/WA in chunks of up to 100 frames. Our comparison is therefore comparable in length to these established evaluation protocols.
We hope this context helps clarify the rationale for our evaluation. We also appreciate your interest in longer trajectories, and this would be a valuable direction for future work. Thank you again for your constructive feedback.
Remaining concern about metric world-space validation
Thank you for the additional evidence and detailed clarifications. The corrected contact evaluation, direct distance-head results, and consistency statistics substantially address my concerns about the interaction attributes. I no longer consider the previously reported dense-contact degradation and distance errors valid grounds for those criticisms.
My remaining concern is the validation of the final world-space trajectories. I appreciate the clarification that the drift discussion refers to the evaluated two-second windows. However, W and WA both fit a similarity transform within each window, and the camera ATE in Appendix J also uses per-window Sim(3) alignment. These results support short-window reconstruction accuracy, but leave the metric accuracy of the composed world-space trajectories unclear because window-level scale errors can be removed by the alignment.
The raw hand errors and depth results at the model's own scale provide useful evidence. Still, they evaluate camera-space hand geometry and scene depth separately. The final world-space trajectory also depends on the predicted camera motion and, for longer videos, the alignment between clips. The qualitative demonstrations and explanation of the stitching procedure do not yet quantify the accumulated error in that final output.
This is a narrower concern than in my initial review, and I appreciate the substantial improvements in the response. However, it remains important to my assessment of the world-space contribution, so I will maintain my original recommendation.