acceptodds
Under review as a conference paper at ICLR 2027

EgoHEP: Predictive Hand-Eye Pretraining for Robot Visual Representation Learning

Abstract

Robotic manipulation requires visual representations that support the selection of relevant objects and regions for interaction. A central challenge is to learn from the way humans actively select visual information and coordinate it with subsequent hand movements. We introduce EgoHEP, a predictive hand-eye pretraining framework that learns this relationship from gaze and hand-wrist motion in egocentric recordings. EgoHEP combines hand-wrist reconstruction and forecasting with spatial attention predicted under gaze supervision, then aligns attention-weighted visual features with representations of future hand-wrist trajectories across multiple horizons, linking visual selection to manipulation dynamics. Experiments in simulation and on real robots demonstrate the transferability of visual representations learned through predictive hand–eye supervision, with EgoHEP achieving state-of-the-art performance on multiple downstream tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Egocentric Vision, Hand–Eye Pretraining, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_25EQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoHEP, a hand-eye pretraining framework for robotic manipulation. It combines future hand-wrist motion prediction, gaze-supervised attention, and contrastive learning to improve visual representations. The method is evaluated on several simulation benchmarks and real-world robot tasks, showing some promising improvments.

Critical strengths
  1. I think the motivation is quite intresting. Using human gaze and hand motion to learn better visual representations for robot manipulation is a reasonable idea. The overall method is also easy to understand.

  2. The experiments cover different robot embodiments and manipulation tasks, including real-world evaluations. Some results are quite good, especially on Meta-World. The authors also provide ablation studies to support their design choices, which is a nice strengh.

Critical weaknesses
  1. The novelty is somewhat limited. Most components, including hand motion prediction, gaze-guided attention and contrastive learning, are already well studied. The proposed method mainly combines these existing techniques. I don't see a very clear methodological novelty compared with previous works on human affordance and visual representation learning. More detailed comparisions would be helpful.

  2. The benefit of hand-eye coordination is not fully convincing. In Table 5, adding gaze supervision alone only brings minor improvements, while most gains come from predictive alignment. This makes me wonder whether the improvement actually comes from the proposed hand-eye mechanism or simply additional representation learning. The authors also don't provide enough analysis on whether the predicted attention really focuses on useful interaction regions. More direct evidance is needed.

  3. The experimental validation is still limited. Although several benchmarks are included, the comparison with existing visual pretraining methods is not very comprehensive. The real-world experiments only contain three tasks with 20 trials each, which is a rather small scale. Also, the generalizaiton of the learned attention across different tasks and embodiments is not well analyzed.

Questions for authors

Please see the weakness part.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer 25EQ

We sincerely thank Reviewer 25EQ for the constructive feedback, which has helped us clarify our contribution and strengthen the experimental evaluation. In response, we provide additional baseline comparisons, component ablations, quantitative attention analyses, and an expanded real-world evaluation.

W1. Novelty and comparison with related methods

W1: Need a clear methodological novelty compared with previous works on human affordance and visual representation learning. More detailed comparisions would be helpful.

We appreciate the reviewer’s suggestion to clarify the methodological contribution and provide more detailed comparisons. EgoHEP jointly models gaze-supervised visual selection with predictive hand–wrist alignment: the attention-pooled visual representation supervised by gaze is also trained to align with future motion. This shared representation links where the model attends with what the hands will do next. The following comparison summarizes how this jointly modeling differs from related approaches.

MethodSupervisionGazePrediction or alignment targetConnection between vision and motion
OCTEgocentric video–2D hand trajectories and interaction hotspotsForecasting model
HRPEgo4D with detector-derived labels–Future wrist, contact, and active-object locationsRegression heads
MPIEgo4D clips–Interaction keyframes and language-grounded objectsPredictive decoder
MCRRobot data (DROID)–Robot proprioception and action chunksContrastive alignment of global features
EgoHEPAria egocentric recordings✓Bilateral 3D hand–wrist trajectoriesContrastive alignment of gaze-supervised, attention-pooled features with future motion

The methodological distinction lies in using gaze to shape the visual representation that participates in future-motion alignment, so that visual selection and motion prediction are learned jointly.

We added HRP and MCR under the same downstream evaluation protocol. These comparisons examine EgoHEP’s performance relative to existing interaction-supervised and visual–motion representation learning methods. The mean in Table 1 weights the four benchmarks equally.

Table 1. Simulation success rates (%).

MethodFranka-KitchenMeta-WorldAdroit HandRoboTwinMean
MPI-Base59.638.846.066.552.7
HaWoR60.444.856.573.058.7
HRP27.640.043.072.045.7
MCR29.670.864.061.056.4
Hand-Wrist Pretraining62.452.456.071.560.6
EgoHEP66.476.862.576.570.6

These comparisons clarify EgoHEP’s relationship to prior interaction-supervised and visual–motion representation learning methods. Its methodological distinction is that gaze supervises the visual selection used for future-motion alignment, linking where the model attends with what the hands will do next. The comparative results support the effectiveness of this design.

AuthorsOpenReview ↗

Reply to Reviewer 25EQ

W2. Hand–eye mechanism and attention analysis

W2: Whether the improvement actually comes from the proposed hand-eye mechanism or simply additional representation learning. Provide more direct evidance and analysis on whether the predicted attention really focuses on useful interaction regions.

We thank the reviewer for highlighting the need to distinguish the contribution of hand–eye jointly modeling from additional predictive learning. We added component ablations and direct attention evaluations to examine both the source of the performance gains and what the learned attention captures.

  • Component ablations. Alignment w/o Gaze retains learned attention pooling and predictive alignment while removing gaze supervision. GT-HW denotes the hand–wrist-only continuation control, distinct from the earlier Hand-Wrist Pretraining model; Gaze-only adds gaze supervision without predictive alignment.

Table 2. Component ablation success rates (%).

VariantHand–wristGazeAlignmentFranka-KitchenMeta-World
GT-HW✓––60.062.8
Gaze-only✓✓–63.664.8
Alignment w/o Gaze✓–✓64.860.8
EgoHEP✓✓✓66.476.8

On Meta-World, alignment without gaze achieves 60.8%, compared with 62.8% for GT-HW, whereas combining alignment with gaze supervision raises success to 76.8%. Gaze supervision alone reaches 64.8%, so neither component independently reproduces the full model’s improvement. EgoHEP also outperforms both component controls on Franka-Kitchen. These results support the benefit of jointly modeling gaze-supervised visual selection with predictive alignment beyond adding predictive learning alone.

  • Human gaze agreement. To examine whether gaze supervision produces meaningful visual attention, we evaluate 83 frames from 13 validation videos with no take overlap with pretraining data. Samples are fixed independently of model outputs, all models use the same frames and gaze labels, and results are averaged equally across videos. NSS(Normalized Scanpath Saliency) measures normalized attention at recorded gaze locations, CC(Correlation Coefficient) measures spatial correlation with gaze target maps, and KL measures divergence between target and predicted attention distributions.

Table 3. Human gaze agreement.

MethodNSS ↑CC ↑KL ↓
Alignment w/o Gaze0.5170.2001.766
Gaze-only3.0480.6710.652
EgoHEP3.0570.6900.595

EgoHEP substantially improves gaze agreement over alignment without gaze, showing that predictive alignment alone does not recover the same gaze-related spatial information. Its NSS and CC are comparable to Gaze-only, with lower KL. Together with the component ablations, these results suggest that EgoHEP’s larger downstream improvement is not simply explained by better gaze prediction, supporting the value of linking gaze-supervised selection to future motion.

AuthorsOpenReview ↗

Reply to Reviewer 25EQ

  • Robot interaction regions. We further evaluate whether the attention transfers to useful interaction regions using 255 shared observations from 15 reference trajectories across five Meta-World tasks. Samples are fixed independently of model outputs, and simulator-derived masks identify manipulated objects or interaction parts. Target mass measures attention probability within these regions; area-normalized mass divides it by the region’s image-area fraction, with uniform attention scoring 1. Results are averaged within trajectories and then equally across tasks.

Table 4. Attention allocation to robot interaction regions.

MethodTarget mass (%) ↑Area-normalized mass (×) ↑
HRP2.8692.652
Alignment w/o Gaze3.2463.167
Gaze-only4.1883.687
EgoHEP (pretrained)4.6894.362
EgoHEP (robot-adapted)7.1709.938

Before robot adaptation, EgoHEP allocates more attention to interaction regions than either component control, providing direct evidence of attention transfer to these robot observations. Adaptation further increases target mass from 4.689% to 7.170% and area-normalized mass from 4.362× to 9.938×. The component controls use pretrained heads, while HRP uses CLS attention as an external reference. These measurements complement the policy ablations: the joint representation improves downstream success and concentrates more attention on interaction regions, with allocation gains after adaptation varying across tasks.

W3. Experimental breadth, real-world scale, and transfer

W3: The comparison with existing visual pretraining methods is not very comprehensive. The real-world experiments is a rather small scale. The generalizaiton of the learned attention across different tasks and embodiments is not well analyzed.

We appreciate the reviewer’s suggestion to broaden the evaluation and examine attention transfer more directly. Alongside the expanded simulation comparison in Table 1, we doubled the number of real-world tasks from three to six and increased the trials per task from 20 to 50. The real-world evaluation therefore increases from 60 to 300 trials per method.

Table 5. Expanded six-task real-world success rates (%, 50 trials per task and method).

MethodPass BananaLift PlatePlace PenGrab RollerHandover BlockStack Bowls ThreeMean
MPI-Base6.022.024.00.060.00.018.7
Hand-Wrist Pretraining56.056.048.00.076.054.048.3
EgoHEP66.080.064.054.086.086.072.7

EgoHEP achieves the highest success rate on all six tasks, averaging 72.7%, compared with 48.3% for Hand-Wrist Pretraining and 18.7% for MPI-Base. The improvement across all six tasks strengthens the real-world evidence beyond the original three-task evaluation.

Attention transfer across tasks and embodiments. We further analyze attention on 255 shared observations across five Meta-World tasks. Area-normalized mass measures attention within simulator-defined interaction regions relative to their image area.

Table 6. Attention allocation across Meta-World tasks (area-normalized mass, ×)↑.

ModelAssemblyBin PickingButton PressDrawer OpenHammerMean
EgoHEP (pretrained)5.4126.5340.1185.9483.7954.362
EgoHEP (robot-adapted)15.61124.0480.0157.9572.0569.938

Pretrained EgoHEP concentrates attention on interaction regions above the uniform baseline in most tasks, supporting attention transfer from human videos to robot observations. Adaptation increases the mean allocation. These measurements provide evidence of attention transfer within Meta-World, while policy results across four simulation benchmarks and six real-world tasks support broader representation transfer across the evaluated tasks and embodiments.

We hope our response adequately addresses your comments. We will incorporate these results into the revised manuscript and welcome further questions and suggestions.

Reviewer_h38VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose EgoHEP, a visual pretraining method using human gaze and future hand-wrist motion from egocentric videos. The main idea is about learning hand-wrist dynamics through reconstruction and prediction, then using gaze to guide visual attention and contrastive learning with future motion. The learned features then tested on 18 simulation tasks and 3 real-world manipulation tasks.

Critical strengths

Interesting motivation of combining human gaze with future hand motion. Seems to be a reasonable way to learn which visual information is more important for manipulation. Evaluation quite extensive, with different robot embodiments and policy architectures. Especially the improvement on Meta-World are impressive, and real-world results also looks promising. No need for gaze or human hand annotations during robot execution. Also nice to see the Stage II ablations, not just comparison with existing methods.

Critical weaknesses

OCT, HRP, and MPI already explored future hand motion, interaction hotspots, and affordance supervision. The proposed gaze-guided predictive alignment is another combination of these ideas, but the paper does not show what makes it fundamentally different or better Table 5 missing a critical ablation, predictive alignment without gaze supervision. this is to show whether the improvement comes from the proposed hand-eye relationship or just adding future-motion contrastive learning. this shows the main contribution of the paper. The paper only shows downstream task performance, but no quantitative evaluation of whether the attention actually follows human gaze or focuses on useful interaction regions. it is unclear what actually learned from gaze pretraining and how much of it remains. The results are quite inconsistent across tasks. Assembly improves, while Hammer drops compared with GT-HW.

Questions for authors

A predictive alignment baseline without gaze supervision, using uniform or learned attention pooling. Show how much improvement actually comes from gaze. Evaluate the learned attention on unseen human videos and robot observations. Show whether it really focuses on interaction regions. Also compare frozen and fine-tuned attention heads. and test whether predictive alignment actually learns temporal relationships. Shuffle the future-motion targets within the same video, and also test with only one prediction horizon. Report the performance differences.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer h38V

We sincerely thank Reviewer h38V for the constructive feedback and specific experimental suggestions. We have added component ablations, quantitative attention evaluations, and temporal controls to better explain the contribution of hand–eye jointly modeling and its behavior across tasks.

W1. Methodological contribution, attention evidence, and task consistency

W1: Do not show what makes EgoHEP fundamentally different or better. Missing a critical ablation, predictive alignment without gaze supervision. Lack quantitative evaluation of whether the attention actually follows human gaze or focuses on useful interaction regions. The ablation results are inconsistent across tasks in meta-world.

We thank the reviewer for prompting us to clarify our methodological contribution and strengthen the supporting experimental evidence. Our contribution lies in jointly modeling gaze-guided visual selection and future hand–wrist alignment within a unified framework: the attention-pooled visual representation supervised by gaze is also trained to align with future motion. This shared representation links where the model attends with what the hands will do next. The following comparison summarizes how this jointly modeling differs from related approaches.

MethodSupervisionGazePrediction or alignment targetConnection between vision and motion
OCTEgocentric video–2D hand trajectories and interaction hotspotsForecasting model
HRPEgo4D with detector-derived labels–Future wrist, contact, and active-object locationsRegression heads
MPIEgo4D clips–Interaction keyframes and language-grounded objectsPredictive decoder
MCRRobot data (DROID)–Robot proprioception and action chunksContrastive alignment of global features
EgoHEPAria egocentric recordings✓Bilateral 3D hand–wrist trajectoriesContrastive alignment of gaze-supervised, attention-pooled features with future motion

We added HRP, MCR, and Global Motion Alignment under the same downstream evaluation protocol. Global Motion Alignment uses only global image–motion contrastive alignment, with all other pretraining settings matched to EgoHEP, providing a direct comparison with a simpler motion-alignment objective. The mean in Table 1 weights the four benchmarks equally.

MethodFranka-KitchenMeta-WorldAdroit HandRoboTwinMean
MPI-Base59.638.846.066.552.7
HaWoR60.444.856.573.058.7
HRP27.640.043.072.045.7
Global Motion Alignment60.052.460.073.561.5
MCR29.670.864.061.056.4
Hand-Wrist Pretraining62.452.456.071.560.6
EgoHEP66.476.862.576.570.6

EgoHEP outperforms Global Motion Alignment on all four benchmarks, raising mean success from 61.5% to 70.6%. Together with the alignment-without-gaze ablation in Q1(a), these results support the contribution of gaze-supervised visual selection jointly modeling with future motion, beyond adding future-motion contrastive learning alone. The attention evaluations and temporal controls in Q1(b–d) further examine what this modeling learns and how it transfers.

We further evaluate human gaze agreement and attention allocation to robot interaction regions in Q1(b), and test the role of temporal correspondence in Q1(d). The task-level analysis in Q1(a) explicitly reports both improvements and declines, clarifying that the overall benefit varies with task requirements.

AuthorsOpenReview ↗

Reply to Reviewer h38V

Q1. Requested more experiments and analysis

Q1: (a)A predictive alignment baseline without gaze supervision, using uniform or learned attention pooling. (b) Evaluate the learned attention on unseen human videos and robot observations. (c) compare frozen and fine-tuned attention heads. and test whether predictive alignment actually learns temporal relationships. (d) Shuffle the future-motion targets within the same video, and also test with only one prediction horizon.

(a) Predictive alignment without gaze supervision and task-level results

We thank the reviewer for suggesting this important control. We added Alignment w/o Gaze, which retains learned attention pooling and predictive alignment while removing gaze supervision. GT-HW denotes the hand–wrist-only continuation control, while Gaze-only adds gaze supervision without predictive alignment.

Table 1. Component ablation success rates (%).

VariantHand–wristGazeAlignmentFranka-KitchenMeta-World
GT-HW✓––60.062.8
Gaze-only✓✓–63.664.8
Alignment w/o Gaze✓–✓64.860.8
EgoHEP✓✓✓66.476.8

On Meta-World, alignment without gaze reaches 60.8%, compared with 62.8% for GT-HW, whereas EgoHEP reaches 76.8%. Gaze supervision alone also falls short of the full model, reaching 64.8%. Thus, neither component alone reproduces the combined model’s improvement, supporting the benefit of jointly modeling gaze-supervised selection with future-motion alignment. Franka-Kitchen shows gain over gaze-free alignment, indicating that the benefit varies across benchmarks.

Table 2. Meta-World ablation success rates (%).

VariantAssemblyBin PickingButton PressDrawer OpenHammerMean
GT-HW2.018.0100.0100.094.062.8
Gaze only40.00.0100.0100.084.064.8
Alignment w/o Gaze2.034.0100.0100.068.060.8
EgoHEP74.036.0100.098.076.076.8

The task-level trade-offs can be understood in the context of the different manipulation requirements. Assembly emphasizes aligning the manipulated object with its target, making object–target spatial relationships particularly relevant. Hammer additionally requires precise tool orientation and controlled contact during the striking motion, so greater attention to interaction regions alone may not translate into better execution.

(b) Quantitative attention on unseen human videos and robot observations

We added two quantitative analyses using fixed samples selected independently of model outputs.

  • For human gaze agreement, we evaluate 83 frames from 13 validation videos with no take overlap with pretraining data. All models use the same frames and gaze labels, and results are averaged equally across videos. Normalized Scanpath Saliency (NSS) measures normalized attention at recorded gaze locations; Correlation Coefficient (CC) measures spatial correlation with gaze target maps, and KL measures the divergence between target and predicted attention distributions

Table 3. Human gaze agreement.

VariantNSS ↑CC ↑KL ↓
Alignment w/o Gaze0.5170.2001.766
Gaze-only3.0480.6710.652
EgoHEP3.0570.6900.595

EgoHEP substantially improves gaze agreement over alignment without gaze, demonstrating that gaze supervision contributes information not recovered by the alignment objective alone. Its NSS and CC are comparable to Gaze-only, with lower KL. Together with the downstream ablations, these results suggest that EgoHEP’s larger policy improvement is not simply explained by better gaze prediction.

AuthorsOpenReview ↗

Reply to Reviewer h38V

  • For robot interaction regions, we evaluate 255 shared observations from 15 reference trajectories across five Meta-World tasks. Simulator-derived masks identify manipulated objects or interaction parts. Target mass measures attention probability within these regions; area-normalized mass divides it by the region’s image-area fraction, with uniform attention scoring 1. Results are averaged within trajectories and then equally across tasks.

Table 4. Attention allocation to robot interaction regions.

MethodTarget mass (%) ↑Area-normalized mass (×) ↑
HRP2.8692.652
Alignment w/o Gaze3.2463.167
Gaze-only4.1883.687
EgoHEP (pretrained)4.6894.362
EgoHEP (robot-adapted)7.1709.938

Before robot adaptation, EgoHEP allocates more attention to interaction regions than either component control, providing direct evidence that the pretrained attention transfers to these robot observations. Adaptation further increases target mass from 4.689% to 7.170% and area-normalized mass from 4.362× to 9.938×. The component controls use pretrained heads, while HRP uses CLS attention as an external reference. On identical images and a common 14×14 grid, pretrained and adapted EgoHEP maps have CC = 0.455 and JS divergence = 0.274, indicating that policy training changes the attention distribution while preserving spatial similarity to the pretrained pattern.

(c) Frozen versus fine-tuned attention heads

We compare freezing the pretrained attention head with allowing it to adapt during downstream policy learning.

Table 5. Frozen versus fine-tuned attention on Meta-World (%).

Attention headAssemblyBin PickingButton PressDrawer OpenHammerMean
Frozen18.00.0100.088.062.053.6
Fine-tuned74.036.0100.098.076.076.8
Difference+56.0+36.00.0+10.0+14.0+23.2

Table 6. Frozen versus fine-tuned attention on Franka-Kitchen (%).

Attention headTurn KnobOpen MicrowaveSlide DoorLightHinge DoorMean
Frozen92.034.0100.076.018.064.0
Fine-tuned88.040.096.076.032.066.4
Difference−4.0+6.0−4.00.0+14.0+2.4

Fine-tuning improves average success by 23.2 percentage points on Meta-World and 2.4 points on Franka-Kitchen. The larger Meta-World gains occur on Assembly and Bin Picking, where the attention analysis also shows increased allocation to interaction regions. These observations support using gaze-pretrained attention as an initialization that can adapt to downstream tasks.

(d) Shuffled future-motion targets and single-horizon alignment

We thank the reviewer for suggesting controls that examine the temporal structure of the alignment objective. Shuffled Motion replaces the alignment target with a trajectory from a separate time interval in the same video, matched by hand side and prediction horizon, while retaining gaze supervision and reconstruction of the original correct trajectory. Single-Horizon Alignment restricts only the alignment branch to H8, leaving the original hand–wrist and reconstruction supervision unchanged.

Table 7. Temporal ablation success rates (%).

VariantFranka-KitchenMeta-World
Single-Horizon Alignment (H8)63.261.2
Shuffled Motion63.258.8
EgoHEP66.476.8

As shown in Table 7, shuffling future-motion targets reduces Meta-World success from 76.8% to 58.8%, while restricting alignment to H8 reduces it to 61.2%. Both controls also reduce Franka-Kitchen success from 66.4% to 63.2%. Since gaze supervision and the other prediction objectives remain unchanged, these results support the importance of aligning observations with their correct future motion and the additional benefit of using multiple prediction horizons.

We hope these additional experiments clarify the contribution of gaze-supervised predictive alignment and the transfer of learned attention to robot tasks. We will incorporate these results into the revised manuscript and welcome further suggestions.

Reviewer_QHwGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoHEP learns visual representations from egocentric videos using hand–wrist motion prediction, gaze supervision, and alignment with future motion. The frozen scene encoder is then transferred to manipulation policies across four simulation benchmarks and three real-world tasks.

Critical strengths
  • The transfer results cover several embodiments and policy architectures. Gains over MPI and hand–wrist pretraining suggest the representation captures information useful for control.
  • The Stage II ablations match initialization, data, and training budget. This makes them more informative than a comparison between checkpoints trained with different amounts of data.
Critical weaknesses
  • The comparison with prior representation learning is incomplete. HRP already uses future hand and interaction supervision, and MCR uses visual–motion alignment. Could the authors include HRP and a simpler motion-alignment objective under the same downstream protocol? MPI is a useful baseline, but does not resolve these overlaps.
  • The ablations do not separate alignment from its dependence on gaze: there is no alignment-without-gaze variant. Could this control and variation across training seeds be reported?
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer QHwG

We sincerely thank Reviewer QHwG for the positive assessment and constructive suggestions, which have helped us strengthen the experimental evaluation. In response, we provide additional baseline comparisons and component ablations across three training seeds.

W1. Comparison with HRP and simpler motion alignment

W1: Could the authors include HRP and a simpler motion-alignment objective under the same downstream protocol? MPI is a useful baseline, but does not resolve these overlaps.

We thank the reviewer for suggesting these additional comparisons. We added HRP, MCR, and Global Motion Alignment under the same downstream evaluation protocol to compare with related representation learning methods and test a simpler alignment alternative. Global Motion Alignment uses only global image–motion contrastive alignment, with all other pretraining settings matched to EgoHEP. Table 1 weights the four benchmarks equally.

Table 1. Simulation success rates (%).

MethodFranka-KitchenMeta-WorldAdroit HandRoboTwinMean
MPI-Base59.638.846.066.552.7
HaWoR60.444.856.573.058.7
HRP27.640.043.072.045.7
Global Motion Alignment60.052.460.073.561.5
MCR29.670.864.061.056.4
Hand-Wrist Pretraining62.452.456.071.560.6
EgoHEP66.476.862.576.570.6

As shown in Table 1, EgoHEP achieves the highest average success rate across the four benchmarks, although MCR performs slightly better on Adroit. EgoHEP consistently outperforms Global Motion Alignment, with an average gain and the largest improvement on Meta-World. Since the remaining pretraining settings are matched, this comparison suggests that the choice of alignment objective contributes to the improvement beyond the training configuration. These findings support the value of jointly modeling gaze-selected visual features with future hand–wrist motion.

W2. Alignment without gaze and variation across seeds

W2: The ablations do not separate alignment from its dependence on gaze: there is no alignment-without-gaze variant. Could this control and variation across training seeds be reported?

We thank the reviewer for suggesting this important control and evaluation across training seeds. We added Alignment w/o Gaze, which retains learned attention pooling and InfoNCE while removing gaze supervision. GT-HW denotes the hand–wrist-only continuation control; Gaze-only adds gaze supervision to this control without predictive alignment. We also expanded the component ablations from the original training seed 125 to three seeds (125, 126, and 127) to examine variability across training runs.

Table 2. Component ablation success rates (%) across three training seeds.

VariantHand–wristGazeAlignmentFranka-KitchenMeta-World
GT-HW✓––61.5 ± 4.563.5 ± 1.2
Gaze-only✓✓–64.5 ± 4.865.1 ± 1.6
Alignment w/o Gaze✓–✓61.2 ± 5.257.9 ± 3.8
EgoHEP✓✓✓67.7 ± 4.474.0 ± 2.5

Across the three seeds, EgoHEP achieves the highest mean on both. Adding gaze supervision to predictive alignment increases mean success by 6.5 percentage points on Franka-Kitchen and 16.1 points on Meta-World. EgoHEP also exceeds Gaze-only by 3.2 and 8.9 points, respectively, showing that neither component alone reproduces the full model’s improvement. These results support the benefit of jointly modeling gaze-supervised visual selection with predictive alignment beyond adding alignment alone.

We hope these additional experiments clarify our contribution and resolve your concerns. We will include these results in the revised manuscript and welcome any further suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We have carefully addressed the concerns raised by Reviewers QHwG, h38V, and 25EQ through additional baseline comparisons, component and temporal ablations, quantitative attention analyses, and expanded real-world experiments. These new results provide evidence for the effectiveness of jointly modeling gaze-guided visual selection and future hand–wrist alignment, while further clarifying the methodological contribution and transferability of EgoHEP.

Below, we summarize our main rebuttal additions:

1. Expanded baseline comparisons. We added HRP, MCR, and Global Motion Alignment under the same downstream evaluation protocol. EgoHEP achieves the highest average success rate across four simulation benchmarks, outperforming the simpler Global Motion Alignment baseline. (Reviewers QHwG, h38V, and 25EQ)

2. Component ablations and multi-seed evaluation. We introduced an Alignment w/o Gaze baseline to disentangle the contributions of gaze supervision and predictive alignment. We further extended the component ablations to three training seeds. EgoHEP consistently achieves the highest mean performance on both Franka-Kitchen and Meta-World. (Reviewers QHwG, h38V, and 25EQ)

3. Quantitative evaluation of learned attention. We evaluated human gaze agreement on 83 frames from 13 validation videos using NSS, CC, and KL divergence. We also analyzed attention allocation to robot interaction regions using 255 observations across five Meta-World tasks. (Reviewers h38V and 25EQ)

4. Attention adaptation and temporal alignment. We compared frozen and fine-tuned attention heads, finding that fine-tuning improves Meta-World success from 53.6% to 76.8%. We additionally introduced Shuffled Motion and Single-Horizon Alignment controls. (Reviewer h38V)

5. Expanded real-world evaluation. We doubled the number of real-world manipulation tasks from three to six and increased the evaluation trials from 20 to 50 per task, resulting in 300 trials per method. EgoHEP achieves the highest success rate on all six tasks, compared with Hand-Wrist Pretraining and MPI-Base. (Reviewer 25EQ)

We sincerely appreciate the reviewers' valuable suggestions and respectfully thank the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.