- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EgoWeave, a framework for converting heterogeneous egocentric videos into structured motion supervision for robot learning. It uses hand-centered perspective cropping and calibrated coordinate transformations for hand reconstruction, followed by a structured representation of head motion, wrist poses, and hand articulation for human pretraining. Experiments demonstrate improved hand reconstruction accuracy and downstream robot manipulation performance across four simulation suites and one real-world task.
Critical strengths
-
substantially improves reconstruction accuracy under challenging off-axis viewing conditions, supported by controlled experiments across different camera setups.
-
The proposed supervision improves manipulation success across four simulation suites and achieves 52.5% success on a real robot compared to 20.0% with direct finetuning.
Critical weaknesses
-
The reconstruction pipeline builds on existing perspective-cropping techniques, while the individual contributions of the proposed head–wrist–articulation representation are not sufficiently isolated.
-
Equations (10) and (18) appear to omit the future head rotation when transforming wrist-local landmarks into the world frame, raising concerns about the correctness of the compositional loss.
Questions for authors
-
Can the authors provide component-level ablations to isolate the contributions of head motion, head-relative wrist poses, and normalized hand articulation?
-
Can the authors clarify whether Equations (10) and (18) are missing the future head rotation in the landmark composition, and whether the implementation uses the correct transformation?
-
Can the authors provide additional real-world task evaluations or experiments investigating robustness to calibration errors?
5 comments
W2 / Q2: Future-head rotation in Eqs. (10) and (18)
For your question about the future-head rotation, the rotation acts on both the wrist position and the wrist-local landmark offset. We expanded the transform after Eq. (10) in Sec. 4.2 and show the prediction form in Eq. (18) in Appendix A.
Let and . The composed world-frame landmark is
Expanding gives
Since , it follows that . Thus, the outer future-head rotation acts on both the wrist position and the wrist-local landmark offset, so the local offset is rotated by the future world-frame wrist orientation.
Eq. (18) applies the same composition to predicted quantities. We also verified the frozen full-model implementation against the recorded training contract: the model-file SHA256 matches, and a numerical check confirms that world-frame composition applies the future-head rotation.
Revised manuscript. The paragraph immediately following Eq. (10) in Sec. 4.2 (p. 6) expands the landmark transform, and Appendix A (p. 16, Eq. (18)) gives the prediction form.
Q3: Additional physical-robot evidence
Following your suggestion to test another physical task, we evaluated PickPen on the same Franka Research 3 arm and parallel gripper used for Bread Transfer.
PickPen evaluation (Sec. 5.6, Table 9). We compare the same five methods over 20 trials per method. Full success requires placing both pens into the holder; we also report placing at least one pen (including partial completions).
| Method | Full success: 2 pens | At least 1 pen |
|---|---|---|
| Direct finetuning | 0/20 (0%) | 4/20 (20%) |
| EgoDex-style pretraining | 1/20 (5%) | 5/20 (25%) |
| EgoWeave w/ official labels | 1/20 (5%) | 10/20 (50%) |
| EgoWeave w/ wrist–fingertip supervision | 1/20 (5%) | 9/20 (45%) |
| EgoWeave | 4/20 (20%) | 13/20 (65%) |
EgoWeave achieves the highest rate on both metrics, exceeding Direct finetuning by 20 percentage points on full success and 45 percentage points on placing at least one pen. This complements Bread Transfer, where EgoWeave achieves 21/40 (52.5%) versus 8/40 (20.0%) for Direct finetuning (Table 9). Appendix J.2 (Table 24) reports the 0/1/2-pen outcome counts, Fig. 5 shows representative EgoWeave rollouts for both tasks, and Appendix J.1–J.2 (Figs. 8–9) compare rollouts across methods.
Revised manuscript. Sec. 5.6 (p. 10, Table 9, Fig. 5) adds PickPen, and Appendix J.2 (p. 27, Table 24, Fig. 9) details the task setup, trial counts, and rollouts.
W1 / Q1: Prior work and component contributions
Thank you for asking how EgoWeave differs from prior perspective-cropping methods and what each component of our representation contributes. EgoWeave uses calibration to give a frozen hand estimator a consistent view across cameras, then predicts head motion, wrist placement, and finger articulation in their respective frames and supervises them through the 3D hand landmarks they jointly determine.
Observation stage (vs. PCLs and JHands; Secs. 3, 5.2–5.3; Tables 1, 5, 11). PCLs maps pinhole-to-pinhole crops via homography inside a trained network, and JHands describes hand-oriented perspective warping on rectified fisheye images with trained hand models. EgoWeave follows the same hand-oriented perspective idea, but samples each final patch directly from the raw image through the calibrated source projection ΠC in one resampling step and pairs it with crop-consistent 3D decoding. This lets one frozen WiLoR regressor run on monocular/stereo pinhole and fisheye video and export records registered with tracked head poses.
Two controls show what actually drives the gain:
Learning stage and component evidence (Sec. 5.5, Table 7). To test the motion target and training objective, we compared four variants against full EgoWeave:
Full EgoWeave outperforms every tested alternative across all three suites. The cleanest single-factor comparison is removing only the composed-landmark loss while keeping all 153D targets and flow supervision unchanged, which drops success by 8.00 points on DexJoCo, 10.67 on DMG-A, and 14.33 on DMG-B. Replacing the future-head wrist reference with Current Head drops success by 18.00 / 10.00 / 13.67 points, replacing wrist-local articulation with Direct Hand Keypoints drops it by 11.00 / 8.67 / 11.67 points, and removing direct head flow supervision (No Explicit Head) drops it by 9.50 / 3.33 / 6.00 points.
Revised manuscript. Sec. 1 (p. 1) clarifies how reconstructed records supervise pretraining, Appendix D (p. 19, Table 11) reports the B/G/L controls, and Sec. 5.5 (p. 9, Table 7) with Appendix H (p. 24, Tables 18–20) details the component comparisons.