acceptodds
Under review as a conference paper at ICLR 2027

Gaze2Act: Gaze-Guided Vision-Language-Action Policies for Interactive Robot Manipulation

Abstract

Interactive robot manipulation requires specifying which object to manipulate, where to interact with it, and how the target changes during execution. Language instructions alone often leave these aspects of human intent underspecified. We introduce Gaze2Act, a vision-language-action (VLA) framework that incorporates human gaze as a dynamic spatial cue to complement language instructions. Gaze2Act maps first-person gaze into the robot’s view through cross-view semantic matching, yielding an object mask and a gaze point for coarse-to-fine target specification. It integrates these cues into the VLA policy through perception-level prompting and action-level conditioning, enabling precise interactions and target updates during execution. We evaluate Gaze2Act on 16 real-robot tasks spanning seven task categories using a Unitree G1 humanoid. Gaze2Act outperforms the evaluated baselines in intent accuracy and task success rate, with improvements in object disambiguation, fine-grained interaction, and dynamic intent steering. These results support human gaze as a complementary input for specifying spatial intent and guiding VLA policies during interactive manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning, human-robot interaction, gaze-guided control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y2U1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gaze2Act uses human gaze to specify manipulation targets and interaction regions. It maps gaze from the user’s view to the robot’s view and incorporates the resulting cues into a VLA policy through visual prompting and action-level conditioning.

Critical strengths

The problem is practically relevant. The paper includes substantial real-robot experiments, interface comparisons, and ablations supporting the use of both conditioning pathways.

Critical weaknesses

1.Limited methodological novelty. The approach mainly combines existing segmentation, visual matching, and conditioning techniques. The system is useful, but the methodological advance is relatively modest.

2.Transferability remains unclear. Experiments use one robot platform and one VLA backbone, with training performed separately for each task group. The unseen-object experiments are useful, but do not establish whether the approach transfers to new tasks, backbones, or robots without substantial additional training.

3.The source of the improvement is not sufficiently isolated. Gaze2Act receives direct spatial guidance, while the main baselines infer targets from language. Although the ablations provide some evidence, comparisons using the same gaze-derived target information would better establish the benefit of the proposed policy architecture.

Questions for authors

1.How much additional data and training are needed to adapt the system to a new task? 2.Would the gains remain if competing methods received the same gaze-derived target masks? 3.Will implementation code, training configurations, and data preparation and evaluation protocols be made available?

6 comments
AuthorsOpenReview ↗

Response to Reviewer y2U1 (1/2)

Thank you for your thoughtful review and for recognizing the practical relevance of Gaze2Act and our extensive real-robot experiments, interface comparisons, and ablations. Gaze2Act lets users specify target objects and interaction regions through gaze, which resolves the ambiguity of language more directly. To address your questions directly, we conducted an additional experiment during the rebuttal period that gives all methods the same target masks, where Gaze2Act keeps its advantage, and we clarify the adaptation cost and the code release. We have added these to the revised paper, with specific locations provided below.

CW2 / Q1. Transfer and adaptation to new tasks

Our extensive real-robot experiments already validate the generalization of the policy.

Unseen objects. Gaze2Act generalizes to unseen objects without any additional demonstrations or retraining. Ambiguous Obj. and Unseen Obj. share the same training data and differ only in the evaluation objects (Appendix D). For example, the fruit policy trained on apples is evaluated directly on unseen tomatoes, where Gaze2Act reaches 86% Task Success.

New tasks. Adapting Gaze2Act to a new task requires a small set of robot demonstrations for fine-tuning. Even a completely different action needs only 50 demonstrations, as for handing over a cup and pouring water with it (Appendix A.4). Using gaze adds no further data collection. Training masks are annotated directly in the robot view without eye-tracking recordings, and the cross-view grounding is not trained on robot data or scenes and is used zero-shot, so it generalizes to new tasks, objects, and robots. On the policy side, the gaze branch is a lightweight addition to the action head of a pretrained VLA, adding 4.95% parameters with a zero-initialized output that preserves the pretrained action prior, so it is not tied to a specific backbone. We have added this to Appendix A.4 of the revised paper.

CW3 / Q2. Gains with the same target masks

Same target masks. Yes, the gains remain. To compare the policy designs fairly, we gave RoboGround, ControlVLA, and Gaze2Act identical ground-truth target masks on three tasks.

  • Pick Bread Place Bowl, with masks of the target bread and the destination bowl, 25 trials for each of the four bread–bowl combinations.
  • Pour Water, with the cup-handle mask, 25 trials.
  • Handover Cup, with the cup-body mask, 25 trials.

Results. With correct masks, the baselines perform better than in Table 1. Gaze2Act still leads in every row of Table R1 and reaches 92.0% Task Success over 150 trials per method, compared with 68.7% for RoboGround and 73.3% for ControlVLA. RoboGround and ControlVLA are re-implemented on the same GR00T N1.5 backbone as Gaze2Act and trained on the same demonstrations with the same settings (Appendix A.2). Since the backbone, the training data, and the target are all shared, this advantage comes from the policy design itself, which we explain in our response to CW1 below. We have added this comparison to Appendix A.6 of the revised paper (Table 4).

Table R1. Shared oracle-mask experiment, Task Success (%). All three methods receive the ground-truth target mask, and the baselines are also given specific instructions like "the square cream bun" and "the blue bowl". Original rows are from Table 1 (50 trials per task and method). Each oracle row has 25 trials per method, and Pick Bread Place Bowl is split into its four bread–bowl combinations. The last row pools all 150 oracle trials per method.

TaskConditionRoboGroundControlVLAGaze2Act
Pick Bread Place BowlOriginal (Table 1)343494
Pick Bread Place BowlOracle, cream bun → blue bowl767696
Pick Bread Place BowlOracle, cream bun → white bowl647292
Pick Bread Place BowlOracle, croissant → blue bowl767696
Pick Bread Place BowlOracle, croissant → white bowl728092
Pour WaterOriginal (Table 1)526080
Pour WaterOracle, cup handle606884
Handover CupOriginal (Table 1)626888
Handover CupOracle, cup body646892
All three tasksOracle68.773.392.0

The pooled 95% Wilson CIs are 68.7 [60.9, 75.5] for RoboGround and 73.3 [65.7, 79.8] for ControlVLA, both below the 92.0 [86.5, 95.4] of Gaze2Act with no overlap.

Q3. Release of code, configurations, and protocols

Yes. We will release the implementation code, training configurations, data-preparation scripts, and evaluation protocols, as now stated in the reproducibility statement of the revised paper, and the main settings are already described in the appendix.

AuthorsOpenReview ↗

Response to Reviewer y2U1 (2/2)

CW1. Methodological novelty

Method design is not the only focus of this paper. Our main contribution is human gaze as a human-in-the-loop interface for VLA policies. By simply looking, users can specify objects, interaction regions, and destinations and update them during execution, which is hard to express with language alone. To realize this, we study how to use and represent gaze for robot control, compare it with other interfaces in a user study, and build a complete system on a real humanoid.

Our method also differs from the mask-conditioned baselines RoboGround and ControlVLA in three ways.

  • Distinct from their binary masks, Gaze2Act renders the target as visual cues. The outline and gaze heatmap on the image focus the policy on fine-grained part-level regions.
  • Distinct from their mask features, Gaze2Act adds a gaze token with richer target information. Their policies only receive the geometry of the target from a black-and-white mask, whereas the gaze token also encodes the semantic features of the original object, so the policy focuses better on the target it interacts with. The token guides every denoising step through a zero-initialized decoupled cross-attention branch, which preserves the pretrained prior.
  • Distinct from their language-specified targets, Gaze2Act specifies the target through gaze. The target thus no longer depends on grounding a language instruction, which avoids the ambiguity of language.

Although simple, these designs are highly effective. With identical ground-truth masks, Gaze2Act still outperforms RoboGround and ControlVLA by 18.7–23.3 points in Task Success (Table R1), and our ablations in Table 3 and Appendix A.7 confirm each design.

Thank you for the constructive feedback, which helped us clarify both the contribution and the transferability of Gaze2Act. If our responses resolve your concerns, we would sincerely appreciate it if you could consider raising your score.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer y2U1,

As the rebuttal deadline is approaching, could you please have a look at our response? We have clarified the novelty of Gaze2Act, its transfer to new tasks, and its gains under identical target masks, and we will release the code. We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 13968

Reviewer_PFYnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Gaze2Act, a method for fine-tuning VLAs to be conditioned on gaze information for easier object specification and disambiguation. Gaze is utilized in the VLA architecture both as a way of annotating target object and area of interest on the RGB image that goes into the VLM, and also as action-level conditioning through gaze tokens. Experiments demonstrate the effectiveness of the gaze-guided approach over the language-guided approach for traditional VLAs, in terms of success in identifying the intended object and the intended area on the object for grasping.

Critical strengths

The paper is well-motivated and the methodology is sound. The three different stages at which gaze is incorporated within the VLA workflow are well-defined and quite intuitive, and the illustrations and video clearly show that the methodology is effective at the designed tasks. Overall, I think the proposed method makes a meaningful contribution in the space of using multimodal information for VLA instruction and steering.

Critical weaknesses

My main concern is with the experiment design. The authors did not include details on who operated the system for the main results shown in Table 1. The baselines could run autonomously from a fixed instruction, but since Gaze2Act is inherently an interactive method, it is unclear whether the researchers themselves or some recruited participants produced the success rates for the Gaze2Act method. If it were the former, then there could be bias on having familiarity with the system. Moreover, the ablation study could have benefited from a broader scope in task and conditions to better highlight the individual contributions of different gaze augmentation methods. Lastly, the attention analysis at the end of Section 4 raises a concern on whether the environment and the instructions overly favor gaze specification. Figure 5 shows a Pick Bread Place Bowl task where the three pieces of bread are very similar in color and only different in shape, and the instruction for the baseline methods is "Pick cream bun place blue bowl." To me, the "cream bun" is ambiguous and could refer to either bread other than the croissant, which is supported also by the attention visualization for the baseline method, where the two pieces of bread are highlighted approximately equally at the start. This is potentially contradictory to the authors' claim that the baselines receive "the most specific unambiguous language description available." In this particular case, the ambiguity can be further resolved by specifying the shape of the bread, so the provided instruction is not "the most specific unambiguous language description." This is just one particular case but it does invite questions about whether the task design is biased towards gaze.

Questions for authors
  • In Table 1, who operated the Gaze2Act method? Was it some researcher or some recruited participant?
  • Could you report confidence intervals on all values in the table?
  • Other than the bread example I gave above, were there any other baseline language instruction that can be made more specific within the task context?
  • With your baseline language instructions, how many have the target object well-specified through language already, and how many are ambiguous? Does the gap between baseline and gaze decrease when language is well-specified?
5 comments
AuthorsOpenReview ↗

Response to Reviewer PFYn (1/2)

Thank you for your thoughtful review and for recognizing the motivation, the sound methodology, and the contribution of Gaze2Act to multimodal VLA instruction and steering. To address your questions directly, we clarify that the Gaze2Act trials were operated by recruited participants rather than the authors, and we report confidence intervals for all results. We also conducted two additional experiments during the rebuttal period. Gaze2Act remains the best when all methods receive the same ground-truth target masks, and both gaze pathways help on all four tasks of the extended ablation. We have added these to the revised paper, with specific locations provided below.

CW1 / Q1. Fairness of the evaluation participants

The Gaze2Act trials in Table 1 were operated by recruited participants, not by the researchers. Ten participants took part, five with prior robotics experience and five without, and none of them was an author or involved in developing Gaze2Act, so the results do not rely on familiarity with the system. We have added these details to the evaluation protocol in Appendix A.5 of the revised paper. For the interface study, we also recruited ten more participants unrelated to this work, bringing it to 20 participants and strengthening the statistical reliability of the results (Appendix E, Table 13). The interface order was randomized, and participants were not told which interface was ours (Appendix B.2).

Q2. Statistical uncertainty of the results

Yes. We computed 95% confidence intervals for every value in the result tables, using Wilson score intervals for success and accuracy rates and t-intervals over the ten participants for interaction time, workload, and questionnaire ratings. Appendix G gives the intervals for every task in Table 1 (Table 14) and for the other result tables (Tables 15–19). Table R1 reports the intervals for the Table 1 averages.

Table R1. Table 1 averages, % [95% Wilson CI]. Each cell reports Intent Accuracy / Task Success.

AverageVanilla GROOTRoboGroundControlVLAGaze2Act
Object-level (n=500)33.6 [29.6, 37.9] / 24.2 [20.7, 28.1]59.6 [55.2, 63.8] / 42.4 [38.1, 46.8]68.0 [63.8, 71.9] / 46.4 [42.1, 50.8]93.0 [90.4, 94.9] / 89.0 [86.0, 91.5]
Part-level (n=250)40.4 [34.5, 46.6] / 21.2 [16.6, 26.7]67.6 [61.6, 73.1] / 46.4 [40.3, 52.6]70.8 [64.9, 76.1] / 52.0 [45.8, 58.1]80.4 [75.0, 84.8] / 72.4 [66.6, 77.6]
Overall (n=750)35.9 [32.5, 39.4] / 23.2 [20.3, 26.4]62.3 [58.7, 65.7] / 43.7 [40.2, 47.3]68.9 [65.5, 72.1] / 48.3 [44.7, 51.8]88.8 [86.3, 90.9] / 83.5 [80.6, 86.0]

The intervals of Gaze2Act lie above those of every baseline, except for part-level Intent Accuracy against ControlVLA, where they overlap slightly (80.4 [75.0, 84.8] vs. 70.8 [64.9, 76.1]). The corresponding Task Success intervals remain separate (72.4 [66.6, 77.6] vs. 52.0 [45.8, 58.1]).

CW3 / Q3. Fairness of the baseline instructions

Every baseline instruction already names a unique target in its scene by category, color, pattern, or part, such as the blue bowl, the pink patterned cup, or the hammer handle (Table 12). The same holds for the bread scene, where "cream bun" and "croissant" each name one specific type of bread. We understand your concern that a less specific instruction could lower the performance of the baselines. We therefore tested this case in an oracle experiment. In each of the four bread–bowl combinations, the baselines received shape-specific instructions such as "the square cream bun", together with the ground-truth masks of the target bread and bowl. With these inputs, RoboGround rises from 34% to 64–76% and ControlVLA from 34% to 72–80%, while Gaze2Act reaches 92–96% and remains the best in every combination (Table R2). We have also revised Appendix A.2 to state that each baseline instruction names a unique target.

AuthorsOpenReview ↗

Response to Reviewer PFYn (2/2)

Q4. Comparison with well-specified instructions

As noted in Q3, every instruction names a unique target. The gap does narrow when the baselines receive the correct target, most on Pick Bread Place Bowl, but Gaze2Act stays clearly ahead.

The baselines mainly fall short in grounding. Even when shape or color words are added to the instruction, their grounding models still segment the wrong object in many cases. To remove these errors, the oracle experiment gives the baselines ground-truth target masks, on the bread task in Q3 and also on Pour Water and Handover Cup, where the masks cover the cup handle and the cup body. A ground-truth mask gives the baselines an accurate target, so the experiment shows what they can reach with well-specified language.

Over all 150 oracle trials, Gaze2Act reaches 92.0% Task Success, compared with 68.7% for RoboGround and 73.3% for ControlVLA (Table R2). All three policies use the same GR00T N1.5 backbone, demonstrations, and training settings (Appendix A.2), so the remaining gap comes from how each policy uses the target. The baselines condition only on a binary mask, while Gaze2Act highlights the target in the image with an outline and a gaze heatmap and passes its semantic and spatial information to action generation through a gaze token. We have added this comparison to Appendix A.6 of the revised paper (Table 4).

Table R2. Task Success (%) with shared ground-truth target masks. In the oracle rows, all methods receive the ground-truth target mask, and the baselines are prompted with an adjective plus the object name, such as "the square cream bun" and "the blue bowl". Original rows are from Table 1 (50 trials per task and method). Each oracle row has 25 trials per method.

TaskConditionRoboGroundControlVLAGaze2Act
Pick Bread Place BowlOriginal (Table 1)343494
Pick Bread Place BowlOracle, cream bun → blue bowl767696
Pick Bread Place BowlOracle, cream bun → white bowl647292
Pick Bread Place BowlOracle, croissant → blue bowl767696
Pick Bread Place BowlOracle, croissant → white bowl728092
Pour WaterOriginal (Table 1)526080
Pour WaterOracle, cup handle606884
Handover CupOriginal (Table 1)626888
Handover CupOracle, cup body646892
All three tasksOracle68.773.392.0

CW2. Contribution of each gaze pathway on more tasks

We extended the ablation to the two part-level cup tasks with 60 trials per variant (Table R3). On all four tasks, both pathways improve over the baseline and the full model performs best, so the two pathways are complementary. They differ in where they help most. Visual prompting is stronger on the part-level tasks, since the contour and heatmap mark the intended part in the image, while action-level conditioning is stronger on Pick Bread Place Bowl, where the gaze token keeps both the selected bread and the destination bowl explicit throughout denoising. On the cup tasks, action-level conditioning is also strong (42/60 and 44/60), since the DINOv3 features of the handle or body crop already separate the two parts. Zero initialization also improves action-level conditioning over random initialization on every task. We have added these results to Appendix A.7 (Table 5) and point to them from the caption of Table 3.

Table R3. Policy design ablation, successful trials out of 60. Pick Bread Place Bowl and Hammer are from Table 3 (Hammer with 20 trials per part). Pour Water and Handover Cup are new. "Gaze prompting only" uses a contour for Pick Bread Place Bowl and a contour plus gaze heatmap for the part-level tasks.

VariantPick Bread Place BowlHammer (handle/head/neck)Pour WaterHandover Cup
Baseline17/6015/6018/6020/60
Gaze prompting only34/6028/6047/6049/60
Gaze conditioning only (random init)24/6017/6034/6036/60
Gaze conditioning only (zero init)40/6019/6042/6044/60
Gaze2Act (full)55/6039/6052/6054/60

We appreciate your careful attention to the experimental design, which led to the oracle comparison and the extended ablation. We hope these additions address your concerns, and we would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer PFYn,

As the rebuttal deadline is approaching, could you please have a look at our response? We have addressed your concerns about the fairness of the participants and the baseline instructions, the statistical uncertainty of the results, and the ablation on more tasks. We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 13968

Reviewer_bWAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a system that lets a person steer a robot by looking at things. Language commands specify the task; the focus of gaze attention says which object is at play and which part of it the humanoid robot should act on. Across 15 tabletop tasks it reaches 89% intent accuracy, while reference methods (processing language only) reach sub-70%. accuracy. This demonstrates a practical, well-motivated interface. Evaluation is on a real humanoid with dexterous hands, across a broad task set and multiple end-users.

Critical strengths

A working, real-robot demonstration that gaze can specify both object identity and contact location for a pretrained VLA policy. The paper shows gaze conveying object identity, contact location (which part of the hammer to grasp, whether to grip the cup body or handle) and target changes during execution. A modular design whose components are evaluated in isolation: Cross-view grounding is calibration-free and is tested separately over 750 trials under viewpoint change, distractors, occlusion and gaze error, with failure modes identified. The two policy pathways, visual prompting and zero-initialised gaze-token injection, are ablated cleanly and for each a distinct benefit is documented.

Critical weaknesses

The main comparison does not separate the effect of the interface from the effect of the policy design. The baselines are conditioned on language-generated masks ( while Gaze2Act receives gaze-grounded masks. Much of the gap in intent accuracy, the metric where the gains are largest, may therefore reflect the wrong object being segmented rather than any difference in how the policies use spatial conditioning. This matters because the paper claims its contributions as gaze as an interface with specific way of injecting it into the VLA The main results do not establish the second claim against prior mask-conditioned methods.

Questions for authors

See my critical weakness concern: For me, one decisive experiment is missing: RoboGround and ControlVLA given the same gaze-derived mask–point input. Without it, the case for Gaze2Act’s policy design rests on an internal ablation alone, and the novelty largely reduces to integrating off-the-shelf components.

2 comments
AuthorsOpenReview ↗

Response to Reviewer bWAu

Thank you for your thoughtful review and for recognizing Gaze2Act as a practical and well-motivated interface, in which users specify targets and interaction locations through gaze and resolve ambiguity more directly than with language. To address your question directly, we conducted the experiment you suggested during the rebuttal period, and it shows a clear gain from the policy design under identical target masks. We have added it to the revised paper, with the location given below.

CW / Q. Effectiveness of the policy design

To fairly compare the capability of the policy design, we ran an oracle comparison in which RoboGround, ControlVLA, and Gaze2Act receive identical ground-truth target masks on three tasks. Gaze2Act still clearly outperforms both baselines, showing the benefit of the policy design itself.

  • Pick Bread Place Bowl, with masks of the target bread and the destination bowl, 25 trials for each of the four bread–bowl combinations.
  • Pour Water, with the cup-handle mask, 25 trials.
  • Handover Cup, with the cup-body mask, 25 trials.

Separating the two effects. With correct masks, the baselines perform better than in Table 1, which reflects the more accurate target that gaze provides as an interface. A clear gap nevertheless remains. Gaze2Act is the best in every row of Table R1 and reaches 92.0% Task Success over 150 trials per method, compared with 68.7% for RoboGround and 73.3% for ControlVLA. With identical target information, this remaining gap reflects the policy design.

The original results in Table 1 point the same way. Averaged over the 15 tasks, Gaze2Act achieves 88.8% Intent Accuracy and 83.5% Task Success, compared with 62.3% and 43.7% for RoboGround and 68.9% and 48.3% for ControlVLA. The ratio of Task Success to Intent Accuracy shows how often an attempt on the correct target is completed. It is 94.0% for Gaze2Act, compared with 70.2% for RoboGround and 70.0% for ControlVLA.

Why the policy design helps. The two baselines condition only on a binary mask of the target. Gaze2Act turns the same target into richer cues at two levels. At the perception level, the outline and, for part-level tasks, the gaze heatmap highlight the target in the image, so the policy attends to the fine-grained region it should act on. At the action level, a gaze token learns a representation of the semantic and spatial information of the target and guides every denoising step through a zero-initialized decoupled cross-attention branch, which keeps the pretrained action prior. Our ablation in Table 3 confirms that both levels contribute. On Pick Bread Place Bowl, the baseline succeeds in 17/60 trials, visual prompting alone in 34/60, action-level conditioning alone in 40/60, and the full model in 55/60. We have added the oracle comparison to Appendix A.6 of the revised paper (Table 4).

Table R1. Shared oracle-mask experiment, Task Success (%). All three methods receive the ground-truth target mask, and the baselines also receive specific language instructions, such as "the square cream bun" and "the blue bowl". Original rows are from Table 1 (50 trials per task and method). Each oracle row has 25 trials per method, and Pick Bread Place Bowl is split into its four bread–bowl combinations. The last row pools all 150 oracle trials per method.

TaskConditionRoboGroundControlVLAGaze2Act
Pick Bread Place BowlOriginal (Table 1)343494
Pick Bread Place BowlOracle, cream bun → blue bowl767696
Pick Bread Place BowlOracle, cream bun → white bowl647292
Pick Bread Place BowlOracle, croissant → blue bowl767696
Pick Bread Place BowlOracle, croissant → white bowl728092
Pour WaterOriginal (Table 1)526080
Pour WaterOracle, cup handle606884
Handover CupOriginal (Table 1)626888
Handover CupOracle, cup body646892
All three tasksOracle68.773.392.0

The 95% Wilson CIs for the last row are 68.7 [60.9, 75.5] for RoboGround and 73.3 [65.7, 79.8] for ControlVLA, both entirely below 92.0 [86.5, 95.4] for Gaze2Act.

Thank you for suggesting the oracle comparison, which has become the clearest evidence for our policy design. If it resolves your concern, we would greatly appreciate it if you could consider raising your score.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer bWAu,

As the rebuttal deadline is approaching, could you please have a look at our response? Following your suggestion, we ran the oracle mask comparison, which evaluates the policy design under identical ground-truth masks. We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 13968

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ki3A (1/3)

Thank you for your thoughtful review and for recognizing the extensive real-robot evaluation, the ablation that cleanly separates the two gaze pathways, and the practical motivation for an updateable gaze interface. To address your questions directly, we conducted additional experiments during the rebuttal period on the policy design under shared target masks, a larger user study with significance tests, and the matching of identical objects. Gaze2Act stays clearly ahead under shared masks, its usability advantages are significant with 20 participants, and spatial context lets it separate identical objects. We have added these experiments and their details to the revised paper, with specific locations provided below.

CW1 / Q1. Isolating the contribution of the policy design

Yes. To compare our policy design fairly, we ran an oracle comparison in which RoboGround, ControlVLA, and Gaze2Act receive identical ground-truth target masks, so that the target information is the same and free of grounding errors. Gaze2Act remains clearly better than RoboGround and ControlVLA, which isolates the contribution of the policy design.

We evaluated three tasks. In Pick Bread Place Bowl, the masks cover the target bread and the destination bowl, with 25 trials for each of the four bread–bowl combinations. In Pour Water and Handover Cup, they cover the cup handle and the cup body, with 25 trials each.

Results. With correct masks, the baselines do better than in Table 1, which reflects the more accurate target that gaze provides. Gaze2Act is still the best in every row of Table R1, with 92.0% Task Success over all 150 trials per method, compared with 68.7% for RoboGround and 73.3% for ControlVLA. Since the target information is identical, this gap comes from how each policy uses it. RoboGround and ControlVLA condition on a binary target mask, whereas Gaze2Act turns the same target into richer cues at two complementary levels. At the perception level, the target contour is rendered on the RGB observation, so the VLM sees the selected object together with its appearance and surroundings. At the action level, the gaze token encodes the semantic and spatial information of the target and is injected into every DiT block through a zero-initialized decoupled cross-attention branch, keeping the spatial constraint explicit at every denoising step while preserving the pretrained action prior. We have added this comparison to Appendix A.6 of the revised paper (Table 4).

Beyond the policy design, the main advantage of Gaze2Act is that users can interact with the robot naturally through their gaze, which conveys their intent directly to the VLA policy and avoids the ambiguity of language. In Table 1, Gaze2Act reaches 88.8% Intent Accuracy and 83.5% Task Success on average over the 15 tasks, compared with 62.3% and 43.7% for RoboGround and 68.9% and 48.3% for ControlVLA. Task Success divided by Intent Accuracy, which shows how often an attempt on the correct target is completed, is 94.0% for Gaze2Act but only 70.2% and 70.0% for the two baselines.

Table R1. Shared oracle-mask experiment, Task Success (%). All three methods receive the ground-truth target mask, and the baselines additionally receive specific language instructions such as "the square cream bun" and "the blue bowl". Original rows are from Table 1 (50 trials per task and method). Each oracle row has 25 trials per method, and Pick Bread Place Bowl is split into its four bread–bowl combinations. The last row pools all 150 oracle trials per method.

TaskConditionRoboGroundControlVLAGaze2Act
Pick Bread Place BowlOriginal (Table 1)343494
Pick Bread Place BowlOracle, cream bun → blue bowl767696
Pick Bread Place BowlOracle, cream bun → white bowl647292
Pick Bread Place BowlOracle, croissant → blue bowl767696
Pick Bread Place BowlOracle, croissant → white bowl728092
Pour WaterOriginal (Table 1)526080
Pour WaterOracle, cup handle606884
Handover CupOriginal (Table 1)626888
Handover CupOracle, cup body646892
All three tasksOracle68.773.392.0

The 95% Wilson CIs of the pooled row, 68.7 [60.9, 75.5] for RoboGround and 73.3 [65.7, 79.8] for ControlVLA, lie entirely below 92.0 [86.5, 95.4] for Gaze2Act.

AuthorsOpenReview ↗

Response to Reviewer Ki3A (2/3)

CW2 / Q5. User study size, counterbalancing, and significance

Fairness. Yes. For every participant, the order of the four interfaces and the order of the target objects were randomized, and participants were not told which interface was the proposed method. We have added this to the study protocol in Appendix B.2 of the revised paper.

More participants and significance tests. To strengthen the statistical evidence, we recruited ten additional participants, for a total of 20, who followed the same protocol (Table R2). A Friedman test confirms a significant effect of the interface on workload, naturalness, and control (p < 0.001). Pairwise tests with Holm correction (Wilcoxon signed-rank for ratings, Welch t for time, Fisher exact for selections) show that Gaze2Act has significantly lower workload, higher naturalness, and shorter interaction time than all three interfaces (p < 0.001). Its target selection and control are comparable to mouse clicking, while it needs only about one third of the interaction time. We have added the results to Appendix E (Table 13) and the significance tests to Appendix G (Table 20).

Table R2. Interface comparison with 20 participants. Target Selections and Task Progress are computed over 200 indications per interface, with 95% Wilson CIs in brackets. The remaining values are participant means ± sample standard deviations. Workload is an adapted NASA-TLX score on a 1–7 scale, where lower is better. Naturalness and Control use 1–7 scales, where higher is better. * marks a significant difference from Gaze2Act (Holm-adjusted p < 0.001).

InterfaceTarget Selections (%) ↑Task Progress (%) ↑Interaction Time (s) ↓Workload ↓Naturalness ↑Control ↑
Detailed Language58.0 [51.1, 64.6]*45.0 [38.3, 51.9]*2.43 ± 0.38*3.91 ± 0.83*3.70 ± 1.34*3.55 ± 1.00*
Mouse click91.0 [86.2, 94.2]75.0 [68.6, 80.5]5.21 ± 0.67*3.41 ± 0.95*2.00 ± 0.79*5.25 ± 1.29
Finger pointing62.0 [55.1, 68.4]*47.0 [40.2, 53.9]*2.23 ± 0.43*3.63 ± 0.49*5.25 ± 0.72*2.90 ± 0.97*
Gaze2Act87.0 [81.6, 91.0]74.0 [67.5, 79.6]1.73 ± 0.332.03 ± 0.436.15 ± 0.595.20 ± 1.01

Q2. Truly identical objects

We agree that two truly identical objects are hard to distinguish from their own features alone. We address this by introducing spatial context, which refers to the features of the surrounding environment, such as nearby objects and their positions. The robot first estimates the relative orientation of the two views from scene-level context, and then, under the aligned orientation, distinguishes the identical objects using instance-level context and local spatial layout. For the orientation, it compares the region containing all identical objects in the robot view, with and without a horizontal flip, against the same region in the user's view. For the objects, it compares crops that include the neighboring objects of each instance and matches DINOv3 features only within a small local window, which preserves the side and distance of each neighbor. We tested two or three identical paper cups under same-side, 45°, and opposite-side views with 50 trials each, and Gaze2Act matched the gazed cup in 143 of 150 trials (95.3%). We have added the details to Appendix B.5 of the revised paper. We thank the reviewer for raising this question, which has made our method more complete.

Q3. How Intent Accuracy is defined

Intent Accuracy is scored from the robot's behavior with the same rule for all methods (Appendix A.2). A trial counts as correct when the first action attempt clearly moves toward the specified object at the specified location. Vanilla GROOT can fall below uniform chance because it does not always select one of the candidates. Without explicit spatial conditioning, it often cannot ground the instruction to one of the similar objects, so it grasps between two objects or hesitates without grasping, which counts as incorrect. Explicit spatial conditioning reduces this behavior. Table R3 breaks down the Bread example, where only 17 of the 36 incorrect trials grasp the wrong bread, while the others grasp between two breads, do not reach, or hesitate. We have added this breakdown to Appendix A.2 of the revised paper.

Table R3. First-attempt outcomes of Vanilla GROOT on Ambiguous Obj., Bread. Each scene contains three types of bread. The first two rows form Intent Accuracy (14/50, 28%), and the first row is Task Success (9/50, 18%).

First attemptTrials
Correct bread, task succeeded9
Correct bread, task failed5
Wrong bread17
No reach6
Grasp between two breads9
Hesitation to grasp4
Total50
AuthorsOpenReview ↗

Response to Reviewer Ki3A (3/3)

Q4. Clean training masks versus online masks at test time

Gaze2Act is trained on clean robot-view masks (Appendix D), and switching to online masks at test time has little effect on the policy. Even when SAM3 tracking is imperfect and part of the target region is missed, the contour and the gaze token still mark the correct object. We measured this by evaluating the policy, trained in the same way, with ground-truth masks and with online gaze-grounded masks. With ground-truth masks, Gaze2Act achieves 94%, 84%, and 92% Task Success on Pick Bread Place Bowl, Pour Water, and Handover Cup (Table R1). With online gaze grounding and SAM3 tracking, it achieves 94%, 80%, and 88% on the same tasks (Table 1). The difference is at most 4 percentage points, so the policy is insensitive to the shift from offline annotations to online grounding. This comparison is included in Appendix A.6 of the revised paper.

Thank you again for your time and for the many helpful comments, which have clearly improved our paper. We hope these responses and the new experiments address your concerns. If they do, we would be grateful if you would consider raising your score. We are happy to answer any further questions.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer Ki3A,

As the rebuttal deadline is approaching, could you please have a look at our response? We have answered each of your questions, including the contribution of the policy design, the size and significance of the user study, truly identical objects, the definition of Intent Accuracy, and the switch to online masks at test time. We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 13968

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.