acceptodds
Under review as a conference paper at ICLR 2027

EgoWeave: Structured Motion Supervision from Calibrated Egocentric Video

Abstract

Egocentric demonstrations could provide scalable human-motion supervision for robot learning, but camera projection and motion reference frames make recordings difficult to reuse across devices. We present EgoWeave, which converts heterogeneous egocentric video into structured motion supervision. A hand-centered virtual camera and crop-consistent decoding let one frozen hand estimator process monocular and stereo pinhole or fisheye video. Timestamped reference poses then place the recovered head, wrists, and hand landmarks in a common coordinate system. We organize this motion as head movement, future-head-relative wrists, and normalized wrist-local articulation, and train with a compositional landmark objective before robot-native finetuning. In controlled fixed-WiLoR experiments with annotated hand regions, hand-centered viewing reduces camera-space MPJPE by 52.9% on Aria and 29.8% on Quest 3 in the – off-axis range. In the evaluated short-budget setting, the complete EgoWeave recipe improves task success over direct finetuning by 5.20–20.50 percentage points across four robot suites; under structured supervision, replacing official annotations with EgoWeave's reconstructed motion further improves all four suites. Code is available at https://anonymous.4open.science/r/EgoWeave/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Egocentric video, human motion reconstruction, hand pose estimation, fisheye cameras, robot learning, vision-language-action models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WJnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoWeave, a framework for converting heterogeneous egocentric videos into structured motion supervision for robot learning. It uses hand-centered perspective cropping and calibrated coordinate transformations for hand reconstruction, followed by a structured representation of head motion, wrist poses, and hand articulation for human pretraining. Experiments demonstrate improved hand reconstruction accuracy and downstream robot manipulation performance across four simulation suites and one real-world task.

Critical strengths
  1. substantially improves reconstruction accuracy under challenging off-axis viewing conditions, supported by controlled experiments across different camera setups.

  2. The proposed supervision improves manipulation success across four simulation suites and achieves 52.5% success on a real robot compared to 20.0% with direct finetuning.

Critical weaknesses
  1. The reconstruction pipeline builds on existing perspective-cropping techniques, while the individual contributions of the proposed head–wrist–articulation representation are not sufficiently isolated.

  2. Equations (10) and (18) appear to omit the future head rotation when transforming wrist-local landmarks into the world frame, raising concerns about the correctness of the compositional loss.

Questions for authors
  1. Can the authors provide component-level ablations to isolate the contributions of head motion, head-relative wrist poses, and normalized hand articulation?

  2. Can the authors clarify whether Equations (10) and (18) are missing the future head rotation in the landmark composition, and whether the implementation uses the correct transformation?

  3. Can the authors provide additional real-world task evaluations or experiments investigating robustness to calibration errors?

5 comments
AuthorsOpenReview ↗

W1 / Q1: Prior work and component contributions

Thank you for asking how EgoWeave differs from prior perspective-cropping methods and what each component of our representation contributes. EgoWeave uses calibration to give a frozen hand estimator a consistent view across cameras, then predicts head motion, wrist placement, and finger articulation in their respective frames and supervises them through the 3D hand landmarks they jointly determine.

Observation stage (vs. PCLs and JHands; Secs. 3, 5.2–5.3; Tables 1, 5, 11). PCLs maps pinhole-to-pinhole crops via homography inside a trained network, and JHands describes hand-oriented perspective warping on rectified fisheye images with trained hand models. EgoWeave follows the same hand-oriented perspective idea, but samples each final patch directly from the raw image through the calibrated source projection ΠC\Pi_C in one resampling step and pairs it with crop-consistent 3D decoding. This lets one frozen WiLoR regressor run on monocular/stereo pinhole and fisheye video and export records registered with tracked head poses.

Two controls show what actually drives the gain:

  • Viewing direction vs. resampling path (Appendix D, Table 11). Keeping the global optical axis fixed, global remapping + cropping (B) and direct global sampling (G) produce similar errors (Aria: 61.34 vs. 61.55 mm; Quest 3: 81.02 vs. 82.17 mm). Keeping direct sampling fixed and rotating the virtual camera toward the hand (G to L) reduces MPJPE by 25.4% on Aria (45.89 mm) and 22.6% on Quest 3 (63.60 mm).
  • Frozen reuse on Quest 3 (Sec. 5.3, Table 5). On Quest 3 with GT ROIs, the same frozen WiLoR reaches 12.73 mm root- and scale-aligned hand-shape error versus 52.15 mm for Quest-3-adapted S2DHand (mono) on the same valid-target subset.

Learning stage and component evidence (Sec. 5.5, Table 7). To test the motion target and training objective, we compared four variants against full EgoWeave:

  1. Direct finetuning: no human-motion pretraining.
  2. No Explicit Head: removes direct flow-matching supervision on the nine head-motion channels while keeping current-head conditioning and geometric gradients through future-head predictions.
  3. Current Head: expresses future wrists in the current head frame (with its corresponding decoder) instead of the future head frame.
  4. Direct Hand Keypoints: predicts future landmarks directly in the current head frame at reference scale, applying geometric supervision to those direct landmarks.
  5. Without geometric loss: retains all 153D structured targets and flow supervision, removing only the composed-landmark loss.
MethodDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
No Explicit Head53.0032.0073.33
Current Head44.5025.3365.67
Direct Hand Keypoints51.5026.6767.67
Without geometric loss54.5024.6765.00
EgoWeave62.5035.3379.33

Full EgoWeave outperforms every tested alternative across all three suites. The cleanest single-factor comparison is removing only the composed-landmark loss while keeping all 153D targets and flow supervision unchanged, which drops success by 8.00 points on DexJoCo, 10.67 on DMG-A, and 14.33 on DMG-B. Replacing the future-head wrist reference with Current Head drops success by 18.00 / 10.00 / 13.67 points, replacing wrist-local articulation with Direct Hand Keypoints drops it by 11.00 / 8.67 / 11.67 points, and removing direct head flow supervision (No Explicit Head) drops it by 9.50 / 3.33 / 6.00 points.

Revised manuscript. Sec. 1 (p. 1) clarifies how reconstructed records supervise pretraining, Appendix D (p. 19, Table 11) reports the B/G/L controls, and Sec. 5.5 (p. 9, Table 7) with Appendix H (p. 24, Tables 18–20) details the component comparisons.

AuthorsOpenReview ↗

W2 / Q2: Future-head rotation in Eqs. (10) and (18)

For your question about the future-head rotation, the rotation acts on both the wrist position and the wrist-local landmark offset. We expanded the transform after Eq. (10) in Sec. 4.2 and show the prediction form in Eq. (18) in Appendix A.

Let RH,f=RH(t)ΔRHR_{H,f} = R_H(t)\Delta R_H and pH,f=pH(t)+ΔpHp_{H,f} = p_H(t) + \Delta p_H. The composed world-frame landmark is

Fs,j(a;st)=pH,f+RH,f(ws+Qsd∗ℓˉs,j).F_{s,j}(a; s_t) = p_{H,f} + R_{H,f}\left(w_s + Q_s d^*\bar\ell_{s,j}\right).

Expanding gives

Fs,j(a;st)=pH,f+RH,fws+(RH,fQs)d∗ℓˉs,j.F_{s,j}(a; s_t) = p_{H,f} + R_{H,f}w_s + (R_{H,f}Q_s)d^*\bar\ell_{s,j}.

Since Qs=RH,f⊤Rs(fk)Q_s = R_{H,f}^\top R_s(f_k), it follows that RH,fQs=Rs(fk)R_{H,f}Q_s = R_s(f_k). Thus, the outer future-head rotation acts on both the wrist position and the wrist-local landmark offset, so the local offset is rotated by the future world-frame wrist orientation.

Eq. (18) applies the same composition to predicted quantities. We also verified the frozen full-model implementation against the recorded training contract: the model-file SHA256 matches, and a numerical check confirms that world-frame composition applies the future-head rotation.

Revised manuscript. The paragraph immediately following Eq. (10) in Sec. 4.2 (p. 6) expands the landmark transform, and Appendix A (p. 16, Eq. (18)) gives the prediction form.

AuthorsOpenReview ↗

Q3: Additional physical-robot evidence

Following your suggestion to test another physical task, we evaluated PickPen on the same Franka Research 3 arm and parallel gripper used for Bread Transfer.

PickPen evaluation (Sec. 5.6, Table 9). We compare the same five methods over 20 trials per method. Full success requires placing both pens into the holder; we also report placing at least one pen (including partial completions).

MethodFull success: 2 pensAt least 1 pen
Direct finetuning0/20 (0%)4/20 (20%)
EgoDex-style pretraining1/20 (5%)5/20 (25%)
EgoWeave w/ official labels1/20 (5%)10/20 (50%)
EgoWeave w/ wrist–fingertip supervision1/20 (5%)9/20 (45%)
EgoWeave4/20 (20%)13/20 (65%)

EgoWeave achieves the highest rate on both metrics, exceeding Direct finetuning by 20 percentage points on full success and 45 percentage points on placing at least one pen. This complements Bread Transfer, where EgoWeave achieves 21/40 (52.5%) versus 8/40 (20.0%) for Direct finetuning (Table 9). Appendix J.2 (Table 24) reports the 0/1/2-pen outcome counts, Fig. 5 shows representative EgoWeave rollouts for both tasks, and Appendix J.1–J.2 (Figs. 8–9) compare rollouts across methods.

Revised manuscript. Sec. 5.6 (p. 10, Table 9, Fig. 5) adds PickPen, and Appendix J.2 (p. 27, Table 24, Fig. 9) details the task setup, trial counts, and rollouts.

Reviewer_SuShOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a pipeline, named EgoWeave for converting heterogeneous egocentric videos into structured motion supervision, that can be used for structured human pretraining and native robot adaptation. The pipeline has 2 key contributions. First, they design an observation-to-record interface that maps pinhole and fisheye observations to a hand-centered perspective representation, which allows 3D hand reconstruction with a frozen estimator. Second, they decompose the reconstructed trajectory to head motion, future- head-relative wrist poses, and normalized wrist-local articulation and use it as target for human pretraining. Evaluations are conducted regarding the hand reconstruction accuracy and human-to-robot transfer. They show that hand-centered sampling improves the reconstruction accuracy and the human pre-training improves the success rate of robot-native finetuning on different embodiments.

Critical strengths
  1. The motivation for using large-scale human egocentric data for robot learning is very meaningful and has good application potential.
  2. Tackling heterogeneous cameras and coordinate systems is a real problem for gathering data from different sources, and the proposed pipeline is useful.
  3. The evaluation across different robot embodiments and different tasks shows improvement compared to direct finetuning.
Critical weaknesses
  1. The hand reconstruction pipeline is built with existing models and techniques. PCLs [1] uses a similar geometry-aware perspective crop. The pretrained hand estimator is from WiLoR [2]. JHands[3] introduces hand-oriented local views. Thus it lacks significant technical contribution.
  2. The reconstruction experiments use H2O, the Aria and Quest 3 subsets of HOT3D, but these data do not occur in the robot learning experiments, making the whole pipeline evaluation incomplete.
  3. For the robot learning results, as suggested in Appendix D, 'The current draft reports aggregate snapshot values; paired rollout intervals and independent-checkpoint variation are not reported here'. The variation of robot performance can vary a lot with different seeds, which can probably cover the margin of improvement. The reported results could be unreliable.

[1] PCLs: Geometry-Aware Neural Reconstruction of 3D Pose With Perspective Crop Layers (CVPR 2021) [2] WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild (CVPR 2025) [3] Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects (ECCV 2024)

Questions for authors
  1. How similar are the tasks for human pretraining compared to the robot learning tasks? Are the improvements mainly on tasks with similar actions or target objects? How well can it transfer to other tasks?
  2. As reported in Table 6, EgoWeave with official labels does not perform as well as Egoweave. Can you explain a bit more about the reason behind it?
  3. Why only EgoDex data is used for the human-to-robot transfer experiments? Is it possible to add more datasets to show the ability to process heterogeneous data?
7 comments
AuthorsOpenReview ↗

W1: Technical contribution and component evidence

Thank you for asking us to clarify the technical contribution beyond PCLs, JHands, and WiLoR. EgoWeave uses calibration to give a frozen hand estimator a consistent view across cameras, then predicts head motion, wrist placement, and finger articulation in their respective frames and supervises them through the 3D hand landmarks they jointly determine.

Observation stage (vs. PCLs and JHands; Secs. 3, 5.2–5.3; Tables 1, 5, 11). PCLs maps pinhole-to-pinhole crops via homography inside a trained network, and JHands describes hand-oriented perspective warping on rectified fisheye images with trained hand models. EgoWeave follows the same hand-oriented perspective idea, but samples each final patch directly from the raw image through the calibrated source projection ΠC\Pi_C in one resampling step and pairs it with crop-consistent 3D decoding. This lets one frozen WiLoR regressor run on monocular/stereo pinhole and fisheye video and export records registered with tracked head poses.

Two controls show what actually drives the gain:

  • Viewing direction vs. resampling path (Appendix D, Table 11). Keeping the global optical axis fixed, global remapping + cropping (B) and direct global sampling (G) produce similar errors (Aria: 61.34 vs. 61.55 mm; Quest 3: 81.02 vs. 82.17 mm). Keeping direct sampling fixed and rotating the virtual camera toward the hand (G to L) reduces MPJPE by 25.4% on Aria (45.89 mm) and 22.6% on Quest 3 (63.60 mm).
  • Frozen reuse on Quest 3 (Sec. 5.3, Table 5). On Quest 3 with GT ROIs, the same frozen WiLoR reaches 12.73 mm root- and scale-aligned hand-shape error versus 52.15 mm for Quest-3-adapted S2DHand (mono) on the same valid-target subset.

Learning stage and component evidence (Sec. 5.5, Table 7). To test the motion target and training objective, we compared four variants against full EgoWeave:

  1. Direct finetuning: no human-motion pretraining.
  2. No Explicit Head: removes direct flow-matching supervision on the nine head-motion channels while keeping current-head conditioning and geometric gradients through future-head predictions.
  3. Current Head: expresses future wrists in the current head frame (with its corresponding decoder) instead of the future head frame.
  4. Direct Hand Keypoints: predicts future landmarks directly in the current head frame at reference scale, applying geometric supervision to those direct landmarks.
  5. Without geometric loss: retains all 153D structured targets and flow supervision, removing only the composed-landmark loss.
MethodDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
No Explicit Head53.0032.0073.33
Current Head44.5025.3365.67
Direct Hand Keypoints51.5026.6767.67
Without geometric loss54.5024.6765.00
EgoWeave62.5035.3379.33

Full EgoWeave outperforms every tested alternative across all three suites. The cleanest single-factor comparison is removing only the composed-landmark loss while keeping all 153D targets and flow supervision unchanged, which drops success by 8.00 points on DexJoCo, 10.67 on DMG-A, and 14.33 on DMG-B. Replacing the future-head wrist reference with Current Head drops success by 18.00 / 10.00 / 13.67 points, replacing wrist-local articulation with Direct Hand Keypoints drops it by 11.00 / 8.67 / 11.67 points, and removing direct head flow supervision (No Explicit Head) drops it by 9.50 / 3.33 / 6.00 points.

Revised manuscript. Sec. 1 (p. 1) clarifies how reconstructed records supervise pretraining, Appendix D (p. 19, Table 11) reports the B/G/L controls, and Sec. 5.5 (p. 9, Table 7) with Appendix H (p. 24, Tables 18–20) details the component comparisons.

AuthorsOpenReview ↗

W2 / Q3: HOT3D, H2O, and another pretraining dataset

You asked why we use different datasets for reconstruction and pretraining, and whether the pipeline transfers from a second video source. We clarify the role of each dataset and report pretraining results on EgoQuest.

Dataset roles (Sec. 5.1). HOT3D and H2O serve as geometric reconstruction benchmarks (Secs. 5.1–5.3); in our setup, their available first-person training subsets total approximately 4.5 hours of candidate footage. Policy pretraining instead uses either the 100-hour EgoDex cohort (which also supports the paired annotation-source comparison on shared RGB in Sec. 5.4) or over 100 hours of self-collected EgoQuest video.

EgoQuest pretraining and transfer (Secs. 5.1, 5.5; Table 8). EgoQuest consists of over 100 hours of egocentric video we recorded with Quest headsets, without dataset-provided motion labels. EgoWeave reconstructs motion from these videos into structured pretraining targets, followed by robot-native finetuning on DexJoCo, DexMimicGen A, and DexMimicGen B:

Method / human sourceDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
EgoWeave / EgoQuest58.0035.3370.33
EgoWeave / EgoDex62.5035.3379.33

These results show that EgoWeave can generate policy-pretraining supervision from a second source of human video without dataset-provided motion annotations. Evaluation details and per-task outcomes are provided in Appendix I, Tables 21–23.

Revised manuscript. Sec. 5.1 (p. 6) specifies dataset roles, and Sec. 5.5 (p. 9, Table 8) with Appendix I (p. 25, Tables 21–23) reports the EgoQuest transfer results.

AuthorsOpenReview ↗

W3: Results from three robot-finetuning seeds

To check robot-finetuning stability, we fixed the human-pretrained checkpoint and trained with three finetuning seeds (42, 43, and 44).

On DMG-B, the three seeds were evaluated under the exact same rollout protocol and achieved 79.33%, 72.00%, and 74.67% (mean 75.33%, sample standard deviation 3.71 percentage points). All three exceeded the reported Direct finetuning reference of 65.00%, with the lowest run still ahead by 7.00 percentage points.

Method / seedDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
EgoWeave / 4262.5035.3379.33
EgoWeave / 4365.0033.3372.00
EgoWeave / 4459.5040.0074.67

For transparency, we also list seeds 42–44 on DexJoCo and DMG-A above, where seeds 43 and 44 used multi-attempt or two-round evaluation rules documented in Appendix I (Tables 21–23). Accordingly, DMG-B serves as the matched-protocol seed check, and we report these values as descriptive variation without inferring a statistical significance test (Appendix C).

Revised manuscript. Table 8 (p. 9) and Sec. 5.5 (p. 10) report the seed results and DMG-B summary statistics, and Appendix I (p. 25, Tables 21–23) gives the per-task breakdown and evaluation protocols.

Reviewer_K3LEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoWeave, a pipeline that turns calibrated egocentric video into motion supervision for robot learning. Hand-centered virtual cameras allow a frozen WiLoR estimator to reconstruct hands from different camera models, while registered poses bring the results into a shared frame. The resulting 153D targets separate head movement, head-relative wrist poses, and normalized wrist-local landmarks; a compositional geometric loss is used for human pretraining, followed by robot-native finetuning. The paper evaluates reconstruction on HOT3D and H2O, transfer on four robot suites, and a Franka bread-transfer task. The reported improvements are promising, but more controlled evidence is needed to identify which components produce the downstream gains.

Critical strengths
  1. The paper is generally well organized and easy to follow.
  2. The figures are well designed. The qualitative results are clearly presented, making it easy to understand and compare the different reconstruction outcomes.
  3. The paper provides simulation results across four suites, a real-robot test, comparisons of annotation sources and supervision recipes, and an additional geometric-loss ablation in the appendix.
Critical weaknesses
  1. The paper states that "the fixed-ROI experiment isolates viewing geometry" (Sec. 5.2). However, the robot-transfer experiments do not compare policies trained on global-axis and hand-centered reconstructions of the same videos. Therefore, it is unclear whether the improved reconstruction accuracy actually leads to better robot performance. A controlled comparison using the same training settings would help establish this connection.
  2. The paper states that "the 48D/153D experiments compare complete supervision recipes rather than the addition of coordinates alone" (Sec. 2). However, these recipes change several factors together, including head motion, wrist reference frames, hand landmarks, normalization, and geometric loss. Although Appendix F evaluates the geometric loss separately, the contributions of the other components remain unclear. More focused ablations would help explain which design choices are responsible for the improvements.
Questions for authors
  1. Have you evaluated robot policies trained with global-axis versus hand-centered reconstructions of the same videos, while keeping the supervision and training settings unchanged?
  2. Can you provide additional ablations to separate the effects of head-motion supervision, future-head-relative wrist coordinates, and normalized hand articulation? The geometric-loss ablation is already included in Appendix F.
  3. How many valid training windows and supervised motion channels are available for the official and EgoWeave-reconstructed annotations? Are the two annotation sources compared using matched valid windows?
5 comments
AuthorsOpenReview ↗

W1 / Q1: Viewing geometry and downstream evidence

Thank you for the feedback on the link between reconstruction geometry and downstream policy performance.

We did not train matched policies on global-axis and hand-centered reconstructions of the same EgoDex videos. With WiLoR fixed, changing only the viewing direction reduced camera-space MPJPE in the 40−60∘40-60^\circ off-axis range by 52.9% on Aria (99.00 to 46.66 mm) and 29.8% on Quest 3 (137.14 to 96.28 mm) (Sec. 5.2, Table 1). With detected inputs, hand-centered viewing also lowered MPJPE from 94.05 to 83.88 mm and raised all-target PCK@50 from 34.41% to 48.53% compared with whole-image undistortion followed by cropping (Table 3).

At the policy level, replacing official annotations with the full EgoWeave pipeline's labels on the same EgoDex RGB recordings improved success by 4.33–8.00 percentage points across all four suites under the same structured target (Sec. 5.4, Table 6). Separately, applying EgoWeave to over 100 hours of self-collected Quest video without official labels (EgoQuest) reached 58.00% / 35.33% / 70.33% on DexJoCo / DMG-A / DMG-B, compared with 42.00% / 27.33% / 65.00% for Direct finetuning (Sec. 5.5, Table 8). Taken together, Tables 1–3 isolate the viewing-direction effect on reconstruction, while Tables 6 and 8 evaluate the end-to-end label-generation and pretraining pipeline.

Revised manuscript. Sec. 5.2 (pp. 6–7, Tables 1–3, Fig. 3) reports the fixed-estimator viewing controls, Sec. 5.4 (pp. 8–9, Table 6) gives the matched-RGB annotation comparison with supervision-availability counts, and Sec. 5.5 (p. 9, Table 8) adds the EgoQuest transfer results.

AuthorsOpenReview ↗

Q2: Head, wrist, articulation, and geometric supervision

To address your question about what each component contributes, we compared four variants against full EgoWeave in Sec. 5.5 (Table 7).

Evaluated variants:

  1. Direct finetuning: no human-motion pretraining.
  2. No Explicit Head: removes direct flow-matching supervision on the nine head-motion channels while retaining current-head conditioning and indirect geometric gradients through future-head predictions.
  3. Current Head: expresses future wrists in the current head frame (with its corresponding decoder) instead of the future head frame.
  4. Direct Hand Keypoints: predicts future landmarks directly in the current head frame at the reference hand scale; head and wrist targets remain, but geometric supervision acts on these direct landmarks rather than composing head, wrist, and local-articulation predictions.
  5. Without geometric loss: retains all structured targets and flow supervision, removing the composed-landmark loss.
  6. EgoWeave: combines explicit head supervision, future-head-relative wrists, normalized wrist-local articulation, and geometric composition.
MethodDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
No Explicit Head53.0032.0073.33
Current Head44.5025.3365.67
Direct Hand Keypoints51.5026.6767.67
Without geometric loss54.5024.6765.00
EgoWeave62.5035.3379.33

The table reports success rates (%). Each comparison below uses full EgoWeave as the reference.

Direct head supervision. No Explicit Head retains current-head conditioning and geometric gradients through the future-head prediction. With direct head supervision, full EgoWeave scores 9.50 points higher on DexJoCo, 3.33 on DMG-A, and 6.00 on DMG-B.

Wrist reference frame. Current Head expresses future wrists relative to the current head pose. EgoWeave uses the future head pose so that viewpoint motion and wrist placement have separate targets. Full EgoWeave scores 18.00 points higher on DexJoCo, 10.00 on DMG-A, and 13.67 on DMG-B.

Hand representation and composition. Direct Hand Keypoints predicts overall landmark positions at the reference hand scale. EgoWeave models articulation in the wrist frame and composes it with the head and wrist predictions at the same reference scale. Full EgoWeave scores 11.00 points higher on DexJoCo, 8.67 on DMG-A, and 11.67 on DMG-B.

Geometric supervision. Both models use the same motion targets and flow-matching supervision. With the composed-landmark loss, full EgoWeave scores 8.00 points higher on DexJoCo, 10.67 on DMG-A, and 14.33 on DMG-B. The loss ties the component predictions to the final landmark positions.

Table 7 supports predicting head motion, wrist pose, and finger articulation separately and combining the predictions for landmark supervision.

Revised manuscript. Sec. 5.5 (p. 9, Table 7) presents the definitions and suite results, and Appendix H (p. 24, Tables 18–20) reports per-task breakdowns and evaluation notes.

AuthorsOpenReview ↗

Q3: Training windows and annotation availability

For your question about how much supervision is available, we counted usable training windows and future steps per channel for the two annotation sources on the same EgoDex RGB recordings.

Valid windows and motion channels (Sec. 5.4, Appendix F, Tables 12–13). The approximately 100-hour EgoDex cohort contains 10.8 million RGB frames and 21.6 million hand-frame slots. The official labels cover all slots, and EgoWeave reconstructs valid hand labels for 19,852,839 (91.91%).

QuantityOfficialReconstructed
Indexed windows4,710,3944,699,960
Head future steps / channel150,732,608150,398,720
Left wrist/articulation steps / channel150,732,608131,589,215
Right wrist/articulation steps / channel150,732,608145,408,501

The reconstructed cache has 99.78% as many indexed windows as the official cache, with 87.30% (left hand) and 96.47% (right hand) of the official valid future steps per channel. We apply the window and channel rules to each annotation source independently. Appendix F (Table 12) details the admission rules and per-channel masks.

Downstream comparison on shared RGB (Sec. 5.4, Table 6). Using the same EgoDex RGB videos and the same structured target (with Direct finetuning shown for reference):

MethodDexJoCoDMG-ADMG-B
Direct finetuning42.0027.3365.00
EgoWeave w/ official labels54.5028.6775.00
EgoWeave w/ reconstructed labels62.5035.3379.33

Reconstructed labels improve transfer over official labels by 8.00 / 6.67 / 4.33 points. Matched RGB comparisons (Sec. 5.4, Fig. 4) illustrate differences in landmark placement and hand geometry between the two sources; together, reconstruction, coordinate registration, and label processing determine the final supervision quality.

Revised manuscript. Sec. 5.4 (pp. 8–9) summarizes supervision availability alongside Table 6 and Fig. 4, and Appendix F (pp. 20–21, Tables 12–13) gives the mask definitions and channel counts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.