- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EgoHEP, a hand-eye pretraining framework for robotic manipulation. It combines future hand-wrist motion prediction, gaze-supervised attention, and contrastive learning to improve visual representations. The method is evaluated on several simulation benchmarks and real-world robot tasks, showing some promising improvments.
Critical strengths
-
I think the motivation is quite intresting. Using human gaze and hand motion to learn better visual representations for robot manipulation is a reasonable idea. The overall method is also easy to understand.
-
The experiments cover different robot embodiments and manipulation tasks, including real-world evaluations. Some results are quite good, especially on Meta-World. The authors also provide ablation studies to support their design choices, which is a nice strengh.
Critical weaknesses
-
The novelty is somewhat limited. Most components, including hand motion prediction, gaze-guided attention and contrastive learning, are already well studied. The proposed method mainly combines these existing techniques. I don't see a very clear methodological novelty compared with previous works on human affordance and visual representation learning. More detailed comparisions would be helpful.
-
The benefit of hand-eye coordination is not fully convincing. In Table 5, adding gaze supervision alone only brings minor improvements, while most gains come from predictive alignment. This makes me wonder whether the improvement actually comes from the proposed hand-eye mechanism or simply additional representation learning. The authors also don't provide enough analysis on whether the predicted attention really focuses on useful interaction regions. More direct evidance is needed.
-
The experimental validation is still limited. Although several benchmarks are included, the comparison with existing visual pretraining methods is not very comprehensive. The real-world experiments only contain three tasks with 20 trials each, which is a rather small scale. Also, the generalizaiton of the learned attention across different tasks and embodiments is not well analyzed.
Questions for authors
Please see the weakness part.
3 comments
Reply to Reviewer 25EQ
W2. Hand–eye mechanism and attention analysis
W2: Whether the improvement actually comes from the proposed hand-eye mechanism or simply additional representation learning. Provide more direct evidance and analysis on whether the predicted attention really focuses on useful interaction regions.
We thank the reviewer for highlighting the need to distinguish the contribution of hand–eye jointly modeling from additional predictive learning. We added component ablations and direct attention evaluations to examine both the source of the performance gains and what the learned attention captures.
- Component ablations. Alignment w/o Gaze retains learned attention pooling and predictive alignment while removing gaze supervision. GT-HW denotes the hand–wrist-only continuation control, distinct from the earlier Hand-Wrist Pretraining model; Gaze-only adds gaze supervision without predictive alignment.
Table 2. Component ablation success rates (%).
| Variant | Hand–wrist | Gaze | Alignment | Franka-Kitchen | Meta-World |
|---|---|---|---|---|---|
| GT-HW | ✓ | – | – | 60.0 | 62.8 |
| Gaze-only | ✓ | ✓ | – | 63.6 | 64.8 |
| Alignment w/o Gaze | ✓ | – | ✓ | 64.8 | 60.8 |
| EgoHEP | ✓ | ✓ | ✓ | 66.4 | 76.8 |
On Meta-World, alignment without gaze achieves 60.8%, compared with 62.8% for GT-HW, whereas combining alignment with gaze supervision raises success to 76.8%. Gaze supervision alone reaches 64.8%, so neither component independently reproduces the full model’s improvement. EgoHEP also outperforms both component controls on Franka-Kitchen. These results support the benefit of jointly modeling gaze-supervised visual selection with predictive alignment beyond adding predictive learning alone.
- Human gaze agreement. To examine whether gaze supervision produces meaningful visual attention, we evaluate 83 frames from 13 validation videos with no take overlap with pretraining data. Samples are fixed independently of model outputs, all models use the same frames and gaze labels, and results are averaged equally across videos. NSS(Normalized Scanpath Saliency) measures normalized attention at recorded gaze locations, CC(Correlation Coefficient) measures spatial correlation with gaze target maps, and KL measures divergence between target and predicted attention distributions.
Table 3. Human gaze agreement.
| Method | NSS ↑ | CC ↑ | KL ↓ |
|---|---|---|---|
| Alignment w/o Gaze | 0.517 | 0.200 | 1.766 |
| Gaze-only | 3.048 | 0.671 | 0.652 |
| EgoHEP | 3.057 | 0.690 | 0.595 |
EgoHEP substantially improves gaze agreement over alignment without gaze, showing that predictive alignment alone does not recover the same gaze-related spatial information. Its NSS and CC are comparable to Gaze-only, with lower KL. Together with the component ablations, these results suggest that EgoHEP’s larger downstream improvement is not simply explained by better gaze prediction, supporting the value of linking gaze-supervised selection to future motion.
Reply to Reviewer 25EQ
- Robot interaction regions. We further evaluate whether the attention transfers to useful interaction regions using 255 shared observations from 15 reference trajectories across five Meta-World tasks. Samples are fixed independently of model outputs, and simulator-derived masks identify manipulated objects or interaction parts. Target mass measures attention probability within these regions; area-normalized mass divides it by the region’s image-area fraction, with uniform attention scoring 1. Results are averaged within trajectories and then equally across tasks.
Table 4. Attention allocation to robot interaction regions.
| Method | Target mass (%) ↑ | Area-normalized mass (×) ↑ |
|---|---|---|
| HRP | 2.869 | 2.652 |
| Alignment w/o Gaze | 3.246 | 3.167 |
| Gaze-only | 4.188 | 3.687 |
| EgoHEP (pretrained) | 4.689 | 4.362 |
| EgoHEP (robot-adapted) | 7.170 | 9.938 |
Before robot adaptation, EgoHEP allocates more attention to interaction regions than either component control, providing direct evidence of attention transfer to these robot observations. Adaptation further increases target mass from 4.689% to 7.170% and area-normalized mass from 4.362× to 9.938×. The component controls use pretrained heads, while HRP uses CLS attention as an external reference. These measurements complement the policy ablations: the joint representation improves downstream success and concentrates more attention on interaction regions, with allocation gains after adaptation varying across tasks.
W3. Experimental breadth, real-world scale, and transfer
W3: The comparison with existing visual pretraining methods is not very comprehensive. The real-world experiments is a rather small scale. The generalizaiton of the learned attention across different tasks and embodiments is not well analyzed.
We appreciate the reviewer’s suggestion to broaden the evaluation and examine attention transfer more directly. Alongside the expanded simulation comparison in Table 1, we doubled the number of real-world tasks from three to six and increased the trials per task from 20 to 50. The real-world evaluation therefore increases from 60 to 300 trials per method.
Table 5. Expanded six-task real-world success rates (%, 50 trials per task and method).
| Method | Pass Banana | Lift Plate | Place Pen | Grab Roller | Handover Block | Stack Bowls Three | Mean |
|---|---|---|---|---|---|---|---|
| MPI-Base | 6.0 | 22.0 | 24.0 | 0.0 | 60.0 | 0.0 | 18.7 |
| Hand-Wrist Pretraining | 56.0 | 56.0 | 48.0 | 0.0 | 76.0 | 54.0 | 48.3 |
| EgoHEP | 66.0 | 80.0 | 64.0 | 54.0 | 86.0 | 86.0 | 72.7 |
EgoHEP achieves the highest success rate on all six tasks, averaging 72.7%, compared with 48.3% for Hand-Wrist Pretraining and 18.7% for MPI-Base. The improvement across all six tasks strengthens the real-world evidence beyond the original three-task evaluation.
Attention transfer across tasks and embodiments. We further analyze attention on 255 shared observations across five Meta-World tasks. Area-normalized mass measures attention within simulator-defined interaction regions relative to their image area.
Table 6. Attention allocation across Meta-World tasks (area-normalized mass, ×)↑.
| Model | Assembly | Bin Picking | Button Press | Drawer Open | Hammer | Mean |
|---|---|---|---|---|---|---|
| EgoHEP (pretrained) | 5.412 | 6.534 | 0.118 | 5.948 | 3.795 | 4.362 |
| EgoHEP (robot-adapted) | 15.611 | 24.048 | 0.015 | 7.957 | 2.056 | 9.938 |
Pretrained EgoHEP concentrates attention on interaction regions above the uniform baseline in most tasks, supporting attention transfer from human videos to robot observations. Adaptation increases the mean allocation. These measurements provide evidence of attention transfer within Meta-World, while policy results across four simulation benchmarks and six real-world tasks support broader representation transfer across the evaluated tasks and embodiments.
We hope our response adequately addresses your comments. We will incorporate these results into the revised manuscript and welcome further questions and suggestions.
Reply to Reviewer 25EQ
We sincerely thank Reviewer 25EQ for the constructive feedback, which has helped us clarify our contribution and strengthen the experimental evaluation. In response, we provide additional baseline comparisons, component ablations, quantitative attention analyses, and an expanded real-world evaluation.
W1. Novelty and comparison with related methods
We appreciate the reviewer’s suggestion to clarify the methodological contribution and provide more detailed comparisons. EgoHEP jointly models gaze-supervised visual selection with predictive hand–wrist alignment: the attention-pooled visual representation supervised by gaze is also trained to align with future motion. This shared representation links where the model attends with what the hands will do next. The following comparison summarizes how this jointly modeling differs from related approaches.
The methodological distinction lies in using gaze to shape the visual representation that participates in future-motion alignment, so that visual selection and motion prediction are learned jointly.
We added HRP and MCR under the same downstream evaluation protocol. These comparisons examine EgoHEP’s performance relative to existing interaction-supervised and visual–motion representation learning methods. The mean in Table 1 weights the four benchmarks equally.
Table 1. Simulation success rates (%).
These comparisons clarify EgoHEP’s relationship to prior interaction-supervised and visual–motion representation learning methods. Its methodological distinction is that gaze supervises the visual selection used for future-motion alignment, linking where the model attends with what the hands will do next. The comparative results support the effectiveness of this design.