- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a novel large-scale dataset (EgoExo4D-Gaze + UMGaze), task, and method/framework (GazeWorld) for 3D world-space gaze target estimation (similar to “gaze following” but in 3D).
GazeWorld leverages 3 frozen foundation models: VGGT-Ω for multi-view 3D reconstruction, DINOv3 for visual feature extraction, and dino.txt for language-alignment of those features. These input features are fused with queries (person-head box embs) and text prompts via so-called “head fusion” and “semantic fusion” modules before passing through a transformer module (with custom attention modules to exchange information between the geometric and semantic input branches), and a final set of task-specific decoders (gaze-heatmap, binary in/out, head-heatmap).
To train the above framework, images from the EgoExo4D dataset are filtered using Gemini 3.8 Flash and converted into a format that permits training, e.g. with in/out labels, 2D heatmaps, head boxes, etc. To allow out-of-domain evaluations, the authors also collect UMGaze, a 200-sample 6-view dataset collected with 5 volunteers.
The authors show that their method is state-of-the-art for multi-view GTE (gaze target estimation) on the MVGT benchmark as well as being competent in both text/bbox-prompted modes on standard single-view GTE benchmarks such as GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept.
Critical strengths
The contribution is very much complete: a new task, training framework, training dataset, OOD test dataset, and evaluations against state-of-the-art methods and VLMs. This is a high-quality paper.
Critical weaknesses
There are no critical weaknesses that I can find in this paper.
Questions for authors
Unconstrained Multi-view Gaze Target Estimation is an interesting task and the proposed GazeWorld architecture is a great first solution for the UM-GTE task.
I note that in Table 1, the 3D metric is at best 0.55 meters. In the scenes that you tested in MVGT, how often does this mean that the incorrect target or instance (if there are many of the same object) is captured? What does 0.55m mean, practically speaking? Similarly, what is “human-level performance” for UM-GTE?
The UMGaze dataset is proposed as an OOD test set that is multi-view. However, it is unclear what makes it OOD. Furthermore, only two metrics are reported on UMGaze (Table 2, right) - why is this the case and what do the missing metrics such as AUC and “3D Metric” look like?
5 comments
Response to Reviewer sLm7 (2/3)
Q2: Human-reference performance
Q2. What is human-level performance on UM-GTE?
Our human-reference pilot yields classification accuracy of 0.910 indoors and 0.926 outdoors, with conditional localization L2 of 0.096 and 0.194. Six raters, each unfamiliar with the annotations, evaluated all 15 six-view groups. Nine indoor groups were assembled by sampling and recombining UMGaze views, yielding 54 views (39 in-frame and 15 out-of-frame); six groups from the outdoor pilot supplied 36 views (13 in-frame and 23 out-of-frame). The recombined indoor groups therefore have a different in-frame distribution from the original benchmark groups.
The views were cropped so that the subject appeared in one to three views, while the target could also appear in views without the subject. Raters saw all six views together with the subject description, head boxes where visible, and an optional VGGT-Ω scene reconstruction. UMGaze supplies no head boxes, so these were annotated for this study; they are also the box prompts in the GazeWorld (text + box) row. They clicked the target in each view where they judged it visible; an unmarked view counted as out of frame. The study was IRB-approved and all raters gave informed consent. All methods are evaluated on the same groups.
In-frame is the positive class, and model decisions use threshold 0.5. Human classification pools six raters’ judgments: 324 indoors and 216 outdoors, with (TP, FP, FN, TN) of (215, 10, 19, 80) and (68, 6, 10, 132), respectively. Conditional L2 is computed on ground-truth in-frame views that each method also classifies as in frame, because humans supply no location for views they judge out of frame. Its evaluation subset therefore varies by method, so we report localization together with classification. The six judgments of each view share the same image evidence.
| Method | Indoor acc ↑ | Indoor F1 ↑ | Indoor conditional L2 ↓ | Outdoor acc ↑ | Outdoor F1 ↑ | Outdoor conditional L2 ↓ |
|---|---|---|---|---|---|---|
| Humans (6 raters, pooled classification) | 0.910 | 0.937 | 0.096 | 0.926 | 0.895 | 0.194 |
| GazeWorld (text + box) | 0.833 | 0.873 | 0.081 | 0.806 | 0.741 | 0.189 |
| GazeWorld (text) | 0.889 | 0.919 | 0.105 | 0.778 | 0.750 | 0.194 |
| Gemini 3.8 Flash (zero-shot, text) | 0.778 | 0.829 | 0.118 | 0.611 | 0.462 | 0.299 |
| Qwen3.8-27B (zero-shot, text) | 0.556 | 0.692 | 0.228 | 0.583 | 0.595 | 0.188 |
| Qwen3.5-9B (SFT, text) | 0.574 | 0.676 | 0.249 | 0.444 | 0.375 | 0.393 |
GazeWorld’s conditional L2 is numerically close to the human reference: 0.081–0.105 versus 0.096 indoors and 0.189–0.194 versus 0.194 outdoors. Human raters achieve higher classification accuracy, especially outdoors: 0.926 versus 0.806 with text-plus-box prompts and 0.778 with text alone. The latter two settings recover 10 and 12 of the 13 in-frame views, respectively, with more false positives under text-only prompting.
This study provides an initial human reference for UM-GTE. The small sample and method-dependent localization subsets support a descriptive comparison, rather than a claim of human-level localization.
Revision. We will add the human-reference protocol and results, including the recombination of indoor views and the definition of conditional localization error.
Response to Reviewer sLm7 (3/3)
Q3: Out-of-domain evaluation
Q3. What makes UMGaze out-of-domain?
UMGaze is out of domain with respect to the task-training data: it contains newly collected scenes and participants, is captured with a different device and acquisition procedure, and is used only for evaluation. No model is trained or tuned on its images. The principal shifts are:
- Handheld capture. The images come from a Google Pixel 10 Pro at 4080×3072. EgoExo4D-Gaze uses exocentric GoPro cameras and a head-mounted Aria device, and MVGT uses fixed calibrated rigs.
- Sequential views. Scene photographs precede participant photographs, with scene structure and target position held fixed. Some views supply scene context without showing the subject.
- New scenes and people. Locations and participants differ from the training data. Half of the samples contain multiple people, so the prompt must identify the intended subject.
The benchmark supplies neither calibration nor head boxes. Table 2 evaluates text prompting without adaptation and averages the four MVGT leave-one-scene-out models (Section 6.4). The participant and camera breakdowns in our response to rFPy are single-model analyses. UMGaze therefore measures transfer to new people, environments, and acquisition conditions under a controlled capture protocol.
Q4: Additional UMGaze metrics
Q4. Why only two metrics on UMGaze, and what do AUC and the 3D metric look like?
GazeWorld’s heatmap AUC on UMGaze is 0.950. Table 2 originally reports L2 and in/out AP because both apply to every compared method. Heatmap AUC additionally applies to GazeWorld; the VLMs return target points rather than heatmaps. We will include AUC with this distinction made explicit.
UMGaze has no 3D target annotations or ground-truth calibration, so the metre-based MVGT metric cannot be computed. We provide a reconstruction-based 3D proxy instead. Annotated and predicted 2D targets are lifted into the same VGGT-Ω reconstruction; we average the distance from each lifted prediction to the consensus of the lifted annotated targets, and normalize by scene scale, the mean distance of reconstructed scene points from their centroid. The VLMs use the same lifting procedure.
| Method | L2 ↓ | In/out AP ↑ | Heatmap AUC ↑ | Reconstruction-based 3D proxy (unitless) ↓ |
|---|---|---|---|---|
| Gemini 3.8 Flash (zero-shot) | 0.366 | 0.772 | N/A | 0.774 |
| Qwen3.8-27B (zero-shot) | 0.284 | 0.863 | N/A | 0.761 |
| Qwen3.5-9B (SFT) | 0.297 | 0.801 | N/A | 0.793 |
| GazeWorld (ours) | 0.095 | 0.973 | 0.950 | 0.240 |
GazeWorld achieves a reconstruction-based proxy error of 0.240, compared with 0.761–0.793 for the VLMs. This supplementary, unitless measure evaluates localization within the shared VGGT-Ω reconstruction, which also supplies GazeWorld’s geometry features. It is distinct from physical 3D accuracy measured against independent ground truth.
Revision. We will add the practical error analysis, clarify the OOD capture conditions, and report UMGaze heatmap AUC and the supplementary 3D proxy.
Response to Reviewer sLm7 (1/3)
Thank you for the positive assessment and the questions about practical accuracy, human performance, and the OOD evaluation.
Q1: Practical meaning of the 3D error
Q1. What does a 0.55 m 3D error mean in practice, and how often is the wrong target or instance chosen?
The 0.546 m value is the average of the four held-out scenes’ median 3D errors. In practical terms, an error of this scale can cross the boundary between nearby objects, so it should be interpreted as spatial localization accuracy rather than object-instance identification. The distance and angular distributions below make its magnitude more concrete.
On 1,617 MVGT frames with an annotated 3D eye position, the median eye-to-target distance is 2.1 m. Median angular error is 9.4° with box prompts and 11.8° with text prompts; 53% of box-prompt predictions fall within 10° and 83% within 20°. The frame-level 3D error distribution places 28% within 0.3 m, 50% within 0.5 m, and 76% within 1.0 m.
Per scene, with box prompts:
The wrong-target or wrong-instance rate has not been quantified. MVGT provides gaze-point annotations without object-instance identities, so this requires an additional instance-level assessment. We will state this distinction explicitly: the present evaluation measures point-localization error, and leaves the frequency of selecting a neighbouring object unresolved.