- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Gaze2Act uses human gaze to specify manipulation targets and interaction regions. It maps gaze from the user’s view to the robot’s view and incorporates the resulting cues into a VLA policy through visual prompting and action-level conditioning.
Critical strengths
The problem is practically relevant. The paper includes substantial real-robot experiments, interface comparisons, and ablations supporting the use of both conditioning pathways.
Critical weaknesses
1.Limited methodological novelty. The approach mainly combines existing segmentation, visual matching, and conditioning techniques. The system is useful, but the methodological advance is relatively modest.
2.Transferability remains unclear. Experiments use one robot platform and one VLA backbone, with training performed separately for each task group. The unseen-object experiments are useful, but do not establish whether the approach transfers to new tasks, backbones, or robots without substantial additional training.
3.The source of the improvement is not sufficiently isolated. Gaze2Act receives direct spatial guidance, while the main baselines infer targets from language. Although the ablations provide some evidence, comparisons using the same gaze-derived target information would better establish the benefit of the proposed policy architecture.
Questions for authors
1.How much additional data and training are needed to adapt the system to a new task? 2.Would the gains remain if competing methods received the same gaze-derived target masks? 3.Will implementation code, training configurations, and data preparation and evaluation protocols be made available?
6 comments
Response to Reviewer y2U1 (2/2)
CW1. Methodological novelty
Method design is not the only focus of this paper. Our main contribution is human gaze as a human-in-the-loop interface for VLA policies. By simply looking, users can specify objects, interaction regions, and destinations and update them during execution, which is hard to express with language alone. To realize this, we study how to use and represent gaze for robot control, compare it with other interfaces in a user study, and build a complete system on a real humanoid.
Our method also differs from the mask-conditioned baselines RoboGround and ControlVLA in three ways.
- Distinct from their binary masks, Gaze2Act renders the target as visual cues. The outline and gaze heatmap on the image focus the policy on fine-grained part-level regions.
- Distinct from their mask features, Gaze2Act adds a gaze token with richer target information. Their policies only receive the geometry of the target from a black-and-white mask, whereas the gaze token also encodes the semantic features of the original object, so the policy focuses better on the target it interacts with. The token guides every denoising step through a zero-initialized decoupled cross-attention branch, which preserves the pretrained prior.
- Distinct from their language-specified targets, Gaze2Act specifies the target through gaze. The target thus no longer depends on grounding a language instruction, which avoids the ambiguity of language.
Although simple, these designs are highly effective. With identical ground-truth masks, Gaze2Act still outperforms RoboGround and ControlVLA by 18.7–23.3 points in Task Success (Table R1), and our ablations in Table 3 and Appendix A.7 confirm each design.
Thank you for the constructive feedback, which helped us clarify both the contribution and the transferability of Gaze2Act. If our responses resolve your concerns, we would sincerely appreciate it if you could consider raising your score.
Looking forward to your reply
Dear Reviewer y2U1,
As the rebuttal deadline is approaching, could you please have a look at our response? We have clarified the novelty of Gaze2Act, its transfer to new tasks, and its gains under identical target masks, and we will release the code. We are looking forward to your reply!
Feel free to let us know if you have any other concerns. Thanks for your time and effort!
Best Regards,
Authors of Submission 13968
Response to Reviewer y2U1 (1/2)
Thank you for your thoughtful review and for recognizing the practical relevance of Gaze2Act and our extensive real-robot experiments, interface comparisons, and ablations. Gaze2Act lets users specify target objects and interaction regions through gaze, which resolves the ambiguity of language more directly. To address your questions directly, we conducted an additional experiment during the rebuttal period that gives all methods the same target masks, where Gaze2Act keeps its advantage, and we clarify the adaptation cost and the code release. We have added these to the revised paper, with specific locations provided below.
CW2 / Q1. Transfer and adaptation to new tasks
Our extensive real-robot experiments already validate the generalization of the policy.
Unseen objects. Gaze2Act generalizes to unseen objects without any additional demonstrations or retraining. Ambiguous Obj. and Unseen Obj. share the same training data and differ only in the evaluation objects (Appendix D). For example, the fruit policy trained on apples is evaluated directly on unseen tomatoes, where Gaze2Act reaches 86% Task Success.
New tasks. Adapting Gaze2Act to a new task requires a small set of robot demonstrations for fine-tuning. Even a completely different action needs only 50 demonstrations, as for handing over a cup and pouring water with it (Appendix A.4). Using gaze adds no further data collection. Training masks are annotated directly in the robot view without eye-tracking recordings, and the cross-view grounding is not trained on robot data or scenes and is used zero-shot, so it generalizes to new tasks, objects, and robots. On the policy side, the gaze branch is a lightweight addition to the action head of a pretrained VLA, adding 4.95% parameters with a zero-initialized output that preserves the pretrained action prior, so it is not tied to a specific backbone. We have added this to Appendix A.4 of the revised paper.
CW3 / Q2. Gains with the same target masks
Same target masks. Yes, the gains remain. To compare the policy designs fairly, we gave RoboGround, ControlVLA, and Gaze2Act identical ground-truth target masks on three tasks.
Results. With correct masks, the baselines perform better than in Table 1. Gaze2Act still leads in every row of Table R1 and reaches 92.0% Task Success over 150 trials per method, compared with 68.7% for RoboGround and 73.3% for ControlVLA. RoboGround and ControlVLA are re-implemented on the same GR00T N1.5 backbone as Gaze2Act and trained on the same demonstrations with the same settings (Appendix A.2). Since the backbone, the training data, and the target are all shared, this advantage comes from the policy design itself, which we explain in our response to CW1 below. We have added this comparison to Appendix A.6 of the revised paper (Table 4).
Table R1. Shared oracle-mask experiment, Task Success (%). All three methods receive the ground-truth target mask, and the baselines are also given specific instructions like "the square cream bun" and "the blue bowl". Original rows are from Table 1 (50 trials per task and method). Each oracle row has 25 trials per method, and Pick Bread Place Bowl is split into its four bread–bowl combinations. The last row pools all 150 oracle trials per method.
The pooled 95% Wilson CIs are 68.7 [60.9, 75.5] for RoboGround and 73.3 [65.7, 79.8] for ControlVLA, both below the 92.0 [86.5, 95.4] of Gaze2Act with no overlap.
Q3. Release of code, configurations, and protocols
Yes. We will release the implementation code, training configurations, data-preparation scripts, and evaluation protocols, as now stated in the reproducibility statement of the revised paper, and the main settings are already described in the appendix.