acceptodds
Under review as a conference paper at ICLR 2027

Learning World-Space Representations of Human Attention from Uncalibrated Views

Abstract

Human gaze target estimation (GTE) remains challenging due to the limited camera field of view. While multi-view setups can mitigate this issue, prior attempts either operate in fixed camera setups or heavily rely on explicit bounding boxes. In this work, we present GazeWorld, a unified framework dedicated to learning world-space representations of human attention directly from uncalibrated multi-view images in the wild. By constructing a unified 3D-aware representation that jointly encodes scene geometry and semantics, GazeWorld leverages pre-trained geometry priors, establishes prompt-based subject association, and predicts per-view gaze targets using features expressed in a shared, estimated 3D coordinate frame. This world-space modeling allows our approach to process a variable number of uncalibrated views, helping resolve ambiguities across views. To support this paradigm, we curated a large-scale dataset from Ego-Exo4D containing over 260K verified multi-view GTE groups, alongside a self-collected in-the-wild benchmark featuring challenging head-invisible scenarios. Experiments show strong GTE performance on multiple benchmarks under both single-view and multi-view scenarios, and demonstrate consistent gains with increasing input views. Code, models, and benchmarks will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-view Gaze Target Estimation, World-Space Representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sLm7OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel large-scale dataset (EgoExo4D-Gaze + UMGaze), task, and method/framework (GazeWorld) for 3D world-space gaze target estimation (similar to “gaze following” but in 3D).

GazeWorld leverages 3 frozen foundation models: VGGT-Ω for multi-view 3D reconstruction, DINOv3 for visual feature extraction, and dino.txt for language-alignment of those features. These input features are fused with queries (person-head box embs) and text prompts via so-called “head fusion” and “semantic fusion” modules before passing through a transformer module (with custom attention modules to exchange information between the geometric and semantic input branches), and a final set of task-specific decoders (gaze-heatmap, binary in/out, head-heatmap).

To train the above framework, images from the EgoExo4D dataset are filtered using Gemini 3.8 Flash and converted into a format that permits training, e.g. with in/out labels, 2D heatmaps, head boxes, etc. To allow out-of-domain evaluations, the authors also collect UMGaze, a 200-sample 6-view dataset collected with 5 volunteers.

The authors show that their method is state-of-the-art for multi-view GTE (gaze target estimation) on the MVGT benchmark as well as being competent in both text/bbox-prompted modes on standard single-view GTE benchmarks such as GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept.

Critical strengths

The contribution is very much complete: a new task, training framework, training dataset, OOD test dataset, and evaluations against state-of-the-art methods and VLMs. This is a high-quality paper.

Critical weaknesses

There are no critical weaknesses that I can find in this paper.

Questions for authors

Unconstrained Multi-view Gaze Target Estimation is an interesting task and the proposed GazeWorld architecture is a great first solution for the UM-GTE task.

I note that in Table 1, the 3D metric is at best 0.55 meters. In the scenes that you tested in MVGT, how often does this mean that the incorrect target or instance (if there are many of the same object) is captured? What does 0.55m mean, practically speaking? Similarly, what is “human-level performance” for UM-GTE?

The UMGaze dataset is proposed as an OOD test set that is multi-view. However, it is unclear what makes it OOD. Furthermore, only two metrics are reported on UMGaze (Table 2, right) - why is this the case and what do the missing metrics such as AUC and “3D Metric” look like?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sLm7 (1/3)

Thank you for the positive assessment and the questions about practical accuracy, human performance, and the OOD evaluation.

Q1: Practical meaning of the 3D error

Q1. What does a 0.55 m 3D error mean in practice, and how often is the wrong target or instance chosen?

The 0.546 m value is the average of the four held-out scenes’ median 3D errors. In practical terms, an error of this scale can cross the boundary between nearby objects, so it should be interpreted as spatial localization accuracy rather than object-instance identification. The distance and angular distributions below make its magnitude more concrete.

On 1,617 MVGT frames with an annotated 3D eye position, the median eye-to-target distance is 2.1 m. Median angular error is 9.4° with box prompts and 11.8° with text prompts; 53% of box-prompt predictions fall within 10° and 83% within 20°. The frame-level 3D error distribution places 28% within 0.3 m, 50% within 0.5 m, and 76% within 1.0 m.

Per scene, with box prompts:

Held-out sceneMedian eye-to-target distance [m]Median 3D error [m] ↓Median angular error [°] ↓
Commons2.850.739.9
Kitchen1.680.318.0
Lab2.710.588.9
Shop1.310.5511.1

The wrong-target or wrong-instance rate has not been quantified. MVGT provides gaze-point annotations without object-instance identities, so this requires an additional instance-level assessment. We will state this distinction explicitly: the present evaluation measures point-localization error, and leaves the frequency of selecting a neighbouring object unresolved.

AuthorsOpenReview ↗

Response to Reviewer sLm7 (2/3)

Q2: Human-reference performance

Q2. What is human-level performance on UM-GTE?

Our human-reference pilot yields classification accuracy of 0.910 indoors and 0.926 outdoors, with conditional localization L2 of 0.096 and 0.194. Six raters, each unfamiliar with the annotations, evaluated all 15 six-view groups. Nine indoor groups were assembled by sampling and recombining UMGaze views, yielding 54 views (39 in-frame and 15 out-of-frame); six groups from the outdoor pilot supplied 36 views (13 in-frame and 23 out-of-frame). The recombined indoor groups therefore have a different in-frame distribution from the original benchmark groups.

The views were cropped so that the subject appeared in one to three views, while the target could also appear in views without the subject. Raters saw all six views together with the subject description, head boxes where visible, and an optional VGGT-Ω scene reconstruction. UMGaze supplies no head boxes, so these were annotated for this study; they are also the box prompts in the GazeWorld (text + box) row. They clicked the target in each view where they judged it visible; an unmarked view counted as out of frame. The study was IRB-approved and all raters gave informed consent. All methods are evaluated on the same groups.

In-frame is the positive class, and model decisions use threshold 0.5. Human classification pools six raters’ judgments: 324 indoors and 216 outdoors, with (TP, FP, FN, TN) of (215, 10, 19, 80) and (68, 6, 10, 132), respectively. Conditional L2 is computed on ground-truth in-frame views that each method also classifies as in frame, because humans supply no location for views they judge out of frame. Its evaluation subset therefore varies by method, so we report localization together with classification. The six judgments of each view share the same image evidence.

MethodIndoor acc ↑Indoor F1 ↑Indoor conditional L2 ↓Outdoor acc ↑Outdoor F1 ↑Outdoor conditional L2 ↓
Humans (6 raters, pooled classification)0.9100.9370.0960.9260.8950.194
GazeWorld (text + box)0.8330.8730.0810.8060.7410.189
GazeWorld (text)0.8890.9190.1050.7780.7500.194
Gemini 3.8 Flash (zero-shot, text)0.7780.8290.1180.6110.4620.299
Qwen3.8-27B (zero-shot, text)0.5560.6920.2280.5830.5950.188
Qwen3.5-9B (SFT, text)0.5740.6760.2490.4440.3750.393

GazeWorld’s conditional L2 is numerically close to the human reference: 0.081–0.105 versus 0.096 indoors and 0.189–0.194 versus 0.194 outdoors. Human raters achieve higher classification accuracy, especially outdoors: 0.926 versus 0.806 with text-plus-box prompts and 0.778 with text alone. The latter two settings recover 10 and 12 of the 13 in-frame views, respectively, with more false positives under text-only prompting.

This study provides an initial human reference for UM-GTE. The small sample and method-dependent localization subsets support a descriptive comparison, rather than a claim of human-level localization.

Revision. We will add the human-reference protocol and results, including the recombination of indoor views and the definition of conditional localization error.

AuthorsOpenReview ↗

Response to Reviewer sLm7 (3/3)

Q3: Out-of-domain evaluation

Q3. What makes UMGaze out-of-domain?

UMGaze is out of domain with respect to the task-training data: it contains newly collected scenes and participants, is captured with a different device and acquisition procedure, and is used only for evaluation. No model is trained or tuned on its images. The principal shifts are:

  1. Handheld capture. The images come from a Google Pixel 10 Pro at 4080×3072. EgoExo4D-Gaze uses exocentric GoPro cameras and a head-mounted Aria device, and MVGT uses fixed calibrated rigs.
  2. Sequential views. Scene photographs precede participant photographs, with scene structure and target position held fixed. Some views supply scene context without showing the subject.
  3. New scenes and people. Locations and participants differ from the training data. Half of the samples contain multiple people, so the prompt must identify the intended subject.

The benchmark supplies neither calibration nor head boxes. Table 2 evaluates text prompting without adaptation and averages the four MVGT leave-one-scene-out models (Section 6.4). The participant and camera breakdowns in our response to rFPy are single-model analyses. UMGaze therefore measures transfer to new people, environments, and acquisition conditions under a controlled capture protocol.

Q4: Additional UMGaze metrics

Q4. Why only two metrics on UMGaze, and what do AUC and the 3D metric look like?

GazeWorld’s heatmap AUC on UMGaze is 0.950. Table 2 originally reports L2 and in/out AP because both apply to every compared method. Heatmap AUC additionally applies to GazeWorld; the VLMs return target points rather than heatmaps. We will include AUC with this distinction made explicit.

UMGaze has no 3D target annotations or ground-truth calibration, so the metre-based MVGT metric cannot be computed. We provide a reconstruction-based 3D proxy instead. Annotated and predicted 2D targets are lifted into the same VGGT-Ω reconstruction; we average the distance from each lifted prediction to the consensus of the lifted annotated targets, and normalize by scene scale, the mean distance of reconstructed scene points from their centroid. The VLMs use the same lifting procedure.

MethodL2 ↓In/out AP ↑Heatmap AUC ↑Reconstruction-based 3D proxy (unitless) ↓
Gemini 3.8 Flash (zero-shot)0.3660.772N/A0.774
Qwen3.8-27B (zero-shot)0.2840.863N/A0.761
Qwen3.5-9B (SFT)0.2970.801N/A0.793
GazeWorld (ours)0.0950.9730.9500.240

GazeWorld achieves a reconstruction-based proxy error of 0.240, compared with 0.761–0.793 for the VLMs. This supplementary, unitless measure evaluates localization within the shared VGGT-Ω reconstruction, which also supplies GazeWorld’s geometry features. It is distinct from physical 3D accuracy measured against independent ground truth.

Revision. We will add the practical error analysis, clarify the OOD capture conditions, and report UMGaze heatmap AUC and the supplementary 3D proxy.

Reviewer_Z6yuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GazeWorld, a prompt‑driven multi‑view gaze target estimation framework which can directly regress head location and gaze location from multiple uncalibrated cameras. Irt mainly utilize frozen foundation encoders to extract meaning geometry representations and attention-based feature fusion module. The authors evaluate the performance of the method on both EgoExo4D‑Gaze and UMGaze with excellent quantitative results.

Critical strengths

(1) Novel problem setting. Considering that common methods focuses on single image gaze target detection. Gaze target detection from uncalibrated views looks like a reasonable and reliable way for real usage.

(2) Solid experimental resulst. The evaluated EgoExo4D‑Gaze dataset provides large‑scale multi‑view gaze annotations combining egocentric and exocentric data, which can benefit the gaze research community for fair evaluation.

Critical weaknesses

(1) The memory cost should be reported considering that this method utilize two large-scale foundation model for feature extraction. And the total parameter sizes can be reported.

(2) Lack of real world demos. Although this paper has achieved remarkable results on academic datasets. I think it will be better to show some visualization results for in-the-wild images.

Questions for authors

See Weaknessess.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Z6yu (1/1)

Thank you for the positive assessment and the suggestions on computational cost and real-world visualizations.

W1: Parameters, memory, and latency

W1. Report memory cost and total parameters, given the two large foundation encoders.

GazeWorld has 2.063B parameters, of which 53.0M are trainable. The frozen encoders account for 2.01B: VGGT-Ω has 1.14B, and dino.txt, including its DINOv3 ViT-L image tower, has 0.87B. Six-view inference takes 256 ms with 8.8 GiB peak GPU memory.

Measurements use one NVIDIA RTX PRO 6000 (Blackwell), one six-view group per evaluation, and 30 measured groups after five warm-up groups. Latency is the median.

MethodTotal paramsTrainable paramsPeak GPU memory [GiB]Latency per 6-view group [ms]
GazeWorld (one joint forward pass)2,063M53.0M8.8256
MVGT baseline (30 two-view evaluations; excludes Metric3D)108M—2.9453
Gaze-LLE (6 single-view evaluations)307M—1.4107
GazeAnywhere (6 single-view evaluations)889M—3.8142
PAGE (6 single-view evaluations)1,697M—7.0474
SAM 3 → RexSeek → SAM 3 (text-to-head-box preprocessing)4,445M—9.37,558

MVGT uses precomputed Metric3D depth maps, so its measurements exclude depth estimation. GazeWorld uses bf16 autocast for its frozen encoders and fp32 for trainable modules; specialized baselines use their released fp32 implementations, while SAM 3 and RexSeek use bf16. The table reports the cost of these implementations under their stated precision settings.

Memory increases from 7.9 to 8.8 GiB as the number of views grows from one to six, while latency increases from 81 to 256 ms:

Input views123456
Peak GPU memory [GiB]7.98.08.18.38.58.8
Latency [ms]81114149173197256

For text-prompt evaluation, GazeWorld accepts the description directly. Box-only baselines additionally run the text-to-head-box pipeline, measured at 7.6 s per group with a separate peak memory of 9.3 GiB. Stage-wise memory peaks are not additive. We will include the parameter counts, memory measurements, and latency results in the paper.

W2: Real-world visualizations

W2. Show visualizations on in-the-wild images.

Figure 2(c), page 9, shows real-world handheld photographs from UMGaze, including scene-only views. It displays ground-truth targets (green crosses), predictions (red circles), heatmaps, and the predicted 3D reconstruction, using text prompts without calibration. These images illustrate how scene context supports gaze localization across views in which the subject is absent.

Our new outdoor pilot extends this evaluation to handheld iPhone captures from two outdoor scenes, mostly at night, with text prompts and no adaptation. It yields L2 0.160 and in/out ROC-AUC 0.882 over 10 groups. We will add qualitative results from this experiment alongside the existing indoor visualization.

Most of my questions are solved. I keep my rating.

We sincerely thank you for the time and thoughtful effort you devoted to reviewing our work and considering our rebuttal. We truly appreciate your constructive engagement throughout the review process and your continued support.

Reviewer_rFPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GazeWorld for gaze target estimation from uncalibrated multi-view images. It combines pretrained geometric and semantic features through 3D-aware fusion and geometric consistency losses, supporting both text and bounding-box prompts. The authors also introduce EgoExo4D-Gaze and UMGaze. The method achieves strong results on existing benchmarks, particularly when the subject's head is not visible.

Critical strengths

1.The paper addresses a relevant limitation of existing multi-view gaze estimation methods by removing the need for supplied camera calibration and pre-associated subject boxes. 2.The experimental results are encouraging. In particular, the controlled ablation in Table 6 provides convincing evidence that the proposed geometric components improve localization beyond what is obtained from pretrained features alone. The additional evaluations under existing multi-view protocols are also appreciated.

Critical weaknesses

1.Incomplete evaluation of world-space estimation. The model ultimately predicts per-view 2D gaze targets, while the reported 3D results rely on ground-truth camera calibration and ground-truth visibility information for triangulation (Appendix D). Although this permits comparison with previous methods, it does not establish the accuracy of fully calibration-free 3D gaze localization. 2.Uncertain dataset quality and selection bias. Approximately 81% of candidate groups are rejected during EgoExo4D-Gaze construction, and the subsequent manual audit focuses on the top 1% of suspected errors. A random independent annotation audit would be useful to assess residual label errors and whether filtering disproportionately removes challenging cases. 3.Limited evidence of in-the-wild generalization. UMGaze contains only 200 samples from five volunteers, with staged target fixation and sequential image capture. The reported improvements are promising, but this evaluation is insufficient to establish robustness across diverse subjects, camera configurations, and less controlled environments. 4.Insufficient evaluation of out-of-frame cases. On EgoExo4D-Gaze, 98.5% of views contain in-frame targets, making the reported in/out AP difficult to interpret. Moreover, the main MVGT in/out AP is calculated on head-visible views. The challenging combination of head invisibility and out-of-frame targets deserves more explicit evaluation.

Questions for authors

1.Can the authors report 3D localization errors using predicted camera geometry and predicted view eligibility, rather than ground-truth calibration and visibility? 2.What is the annotation accuracy on a randomly sampled, independently verified subset of EgoExo4D-Gaze? How does filtering affect the retained data distribution? 3.Can the UMGaze results be broken down by participant and camera configuration, preferably with uncertainty estimates? 4.Can the authors report out-of-frame precision and recall, particularly for views where the subject's head is invisible?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rFPy (1/5)

Thank you for identifying the evaluations needed to clarify the 3D results, dataset quality, and generalization. We address each point below with additional experiments and explicit evaluation protocols.

W1 / Q1: 3D localization with predicted geometry

W1. 3D results rely on ground-truth calibration and visibility; fully calibration-free 3D accuracy is not established.

Using predicted geometry and predicted view eligibility, GazeWorld achieves 0.577 m with box prompts and 0.688 m with text prompts, covering 96.9% and 94.9% of evaluable frames. On frames shared by all settings, the errors are 0.518 / 0.635 m, compared with 0.734 / 0.855 m for the calibrated MVGT baseline. These are Sim(3)-aligned errors under the protocol below.

Setup. We recover the 3D target using the cameras and depth maps predicted by VGGT-Ω and select views using the predicted in-frame probability. Since the reconstruction has an arbitrary coordinate frame and scale, evaluation fits one Sim(3) transform per frame between predicted and ground-truth camera centres and applies it to the recovered target. Ground-truth centres are used only for evaluation alignment; inference receives no calibration. Per-scene median alignment residuals are 3.9–6.0 cm. This measures Sim(3)-aligned 3D localization with calibration-free inference; absolute metric scale requires an external reference.

Coverage. The updated evaluation set contains 2,007 frames with an official 3D target out of 2,292 MVGT test frames: 12,042 evaluated views and 1,710 excluded views. All coverage and common-frame counts below use this set. Triangulation in (a), (b), and the baseline requires at least two usable views; the baseline additionally requires head boxes in those views. Depth-based recovery in (c) and (d) requires at least one usable view. No frame is lost to reconstruction or alignment failure.

Box prompts:

SettingCamera geometryView eligibilityCoverageError, all valid frames [m] ↓Error, common frames [m] ↓
(a) Table 1 protocolGT calibrationGT100%0.5440.462
(b) Predicted eligibilityGT calibrationPredicted (p_in > 0.5)86.8%0.5540.499
(c) Predicted geometryVGGT-ΩGT100%0.5040.451
(d) Predicted geometry and eligibilityVGGT-ΩPredicted (p_in > 0.5)96.9%0.5770.518
MVGT baselineGT calibrationViews with a head box84.6%0.7500.734

Text prompts:

SettingCoverageError, all valid frames [m] ↓Error, common frames [m] ↓
(a)100%0.6750.572
(b)84.6%0.6890.635
(c)100%0.6440.575
(d)94.9%0.6880.635
MVGT baseline84.4%0.8870.855

Errors are medians per held-out scene, averaged equally over the four scenes. Common frames are those where all five rows are valid: 1,530 with box prompts and 1,502 with text prompts. Row (a) re-runs the Table 1 protocol (Table 1: 0.546 / 0.673 m).

On the common frames:

  • The two recovery pipelines achieve similar errors: (c) 0.451 m against (a) 0.462 m with box prompts, and 0.575 m against 0.572 m with text. This compares the complete recovery procedures: calibrated triangulation in (a) and predicted-depth recovery in (c).
  • Predicted eligibility adds 4–7 cm: (a) to (b) and (c) to (d).
  • The fully predicted setting (d) is 0.22 m below the calibrated baseline with both prompt types, and covers more frames (96.9% against 84.6% with box prompts). The baseline's missing frames are mostly in Shop, where 235 of 488 frames have a head box in two views.

Q1. Report 3D errors with predicted camera geometry and predicted view eligibility.

Setting (d) uses both predicted geometry and predicted eligibility. The tables report its accuracy on all valid frames and on the common subset, together with coverage, so the effect of view selection is visible alongside localization error.

Revision. We will add these results and the alignment protocol, update the evaluation counts throughout, and label Table 1 explicitly as using ground-truth calibration and eligibility for 3D evaluation.

AuthorsOpenReview ↗

Response to Reviewer rFPy (2/5)

W2 / Q2: Annotation quality and filtering bias

W2. About 81% of candidates are rejected and the manual audit covers only the top 1%; a random, independent audit is needed.

The random independent audit estimates 94.0% correctness for released groups (95% Wilson CI: 91.6–95.8%), compared with 32.0% for rejected candidates. We sampled 1,000 groups: 500 from each pool. Sampling was proportionally stratified by rejection reason for rejected candidates (338 geometric-check rejections, 62 Gemini-verdict-only rejections, and 100 marked unsure) and by activity for released groups.

Three volunteers who were not involved in building the pipeline each judged all 1,000 groups. Pool membership was hidden through neutral identifiers and per-rater randomized order. Raters saw the egocentric frame with its raw eye-tracker gaze point and every exocentric view with its target annotation or out-of-frame flag. They judged each group as correct, wrong, or cannot tell; the table reports majority judgments, and a single wrong view makes a group wrong. All three raters agreed on 91% of groups (Fleiss' κ = 0.87; pairwise Cohen's κ = 0.81–0.92).

PoolAudited groupsCorrectWrongCorrect rate [95% Wilson CI]
Rejected: Gemini verdict only6219430.306 [0.206, 0.430]
Rejected: geometric checks3381062320.314 [0.266, 0.365]
Rejected: marked unsure10035650.350 [0.264, 0.447]
Rejected: total5001603400.320 [0.281, 0.362]
Released500470300.940 [0.916, 0.958]

The audit measures consistency with the recorded eye-tracker gaze and cross-view visual evidence. It identifies a 6% residual error rate among released groups and shows that filtering also discards usable annotations: 32% of audited rejected groups were judged correct. This quantifies the trade-off between retained-label reliability and data coverage; eye-tracker measurement accuracy remains a separate source of uncertainty.

Q2. Annotation accuracy on a random, independently verified subset, and how filtering shifts the retained distribution.

Under the audit definition above, the estimated correctness of released groups is 94.0% [0.916, 0.958].

Filtering shifts the release toward procedural activities and away from high-motion activities. The following activity shares use the latest statistics following post-submission data auditing and updates. We will revise the corresponding dataset counts and filtering statistics in Appendix F.1 to use this version consistently.

ActivityShare of all candidatesShare of released groups
Health8.4%13.2%
Music11.8%13.9%
Cooking46.0%52.8%
Rock Climbing7.8%7.7%
Bike Repair7.5%6.6%
Basketball6.7%3.6%
Soccer4.2%1.6%
Dance7.6%0.6%

Basketball, soccer, and dance together decrease from 18.5% of candidates to 5.8% of released groups, while cooking increases from 46.0% to 52.8%. Thus, the retained data provide less coverage of high-motion activities. A plausible contributor is the sensitivity of cross-view gaze projection to eye-tracking errors during rapid motion.

Revision. We will add the random audit and activity distribution, synchronize the dataset statistics with the latest audited version, and document both residual label error and reduced coverage of high-motion activities.

AuthorsOpenReview ↗

Response to Reviewer rFPy (3/5)

W3 / Q3: Generalization and UMGaze breakdowns

W3. UMGaze (200 staged samples, five volunteers) is too little evidence of in-the-wild robustness.

We provide participant and camera-configuration breakdowns below, together with a new outdoor pilot. UMGaze evaluates transfer to new participants and independently captured indoor scenes under staged fixation and sequential handheld capture. The outdoor pilot adds 10 groups (60 images) from two scenes captured with an iPhone, mostly at night. All methods use text prompts, and GazeWorld is evaluated without adaptation. These experiments characterize transfer across the sampled conditions; broader outdoor robustness remains to be established.

MethodIndoor L2 ↓Indoor in/out AP ↑Outdoor L2 ↓Outdoor in/out AP ↑
Gemini 3.8 Flash (zero-shot)0.3660.7720.392 [0.239, 0.524]0.301 [0.188, 0.470]
Qwen3.8-27B (zero-shot)0.2840.8630.275 [0.180, 0.408]0.250 [0.133, 0.516]
Qwen3.5-9B (SFT)0.2970.8010.274 [0.167, 0.379]0.241 [0.190, 0.361]
GazeWorld (ours)0.0950.9730.160 [0.122, 0.197]0.661 [0.536, 0.821]

Outdoor L2 is evaluated on 13 in-frame views from six groups; in/out AP uses all 60 views. Brackets denote 95% group-bootstrap intervals, describing uncertainty within this two-scene pilot.

GazeWorld achieves the lowest L2 and highest AP in both settings. Its L2 increases from 0.095 indoors to 0.160 outdoors. The in-frame prevalence also changes from 50% to 21.7% (13/60), so AP should be interpreted within each setting. Outdoors, GazeWorld achieves AP 0.661 and in/out ROC-AUC 0.882 [0.821, 0.931]; VLM AP ranges from 0.241 to 0.301.

Q3. Break UMGaze down by participant and camera configuration, with uncertainty.

By participant, indoor. The following breakdown uses one retrained GazeWorld checkpoint with text prompts, yielding aggregate L2/AP of 0.112 / 0.972. For reference, Table 2 reports 0.095 / 0.973 averaged over four MVGT leave-one-scene-out models. Brackets denote 95% confidence intervals.

Indoor participantGroupsIn-frame / total viewsL2 ↓ [95% CI]In/out AP ↑ [95% CI]
P12060 / 1200.152 [0.061, 0.224]0.944 [0.704, 0.982]
P270210 / 4200.089 [0.033, 0.102]0.980 [0.874, 0.989]
P360180 / 3600.113 [0.078, 0.143]0.981 [0.862, 0.991]
P42060 / 1200.154 [0.021, 0.278]0.911 [0.696, 0.996]
P53090 / 1800.108 [0.033, 0.244]0.942 [0.780, 0.990]
All200600 / 1,2000.112 [0.085, 0.139]0.972 [0.920, 0.981]

L2 ranges from 0.089 to 0.154 and AP from 0.911 to 0.981 across participants. The two participants with 20 groups have the highest L2, and intervals are wide for participants with 20–30 groups.

By participant, outdoor pilot. Intervals resample groups within each participant; each participant has three or four groups.

Outdoor participantGroupsIn-frame / total viewsL2 ↓ [95% CI]In/out AP ↑ [95% CI]
A46 / 240.204 [0.179, 0.248]0.755 [0.666, 1.000]
B34 / 180.150 [0.133, 0.178]0.597 [0.196, 0.875]
C33 / 180.085 [0.071, 0.114]0.819 [0.677, 1.000]

By camera configuration. The indoor data contain 200 groups of six sequential handheld views, captured with a Google Pixel 10 Pro at 4080×3072. We characterize their geometry using camera poses estimated by the frozen VGGT-Ω tower. For each group, we compute the median pairwise angle between the six optical axes and divide groups into tertiles, using geometry alone to define the bins. This breakdown uses one original checkpoint from the set evaluated in Table 2, with aggregate L2/AP of 0.093 / 0.968; Table 2 averages four models. Brackets denote 95% group-bootstrap intervals.

Estimated viewing-angle binAngle rangeGroupsIn-frame / total viewsL2 ↓ [95% CI]In/out AP ↑ [95% CI]
Low30.0°–69.6°67201 / 4020.106 [0.064, 0.159]0.959 [0.798, 0.980]
Mid70.5°–85.0°66198 / 3960.077 [0.038, 0.136]0.974 [0.862, 0.983]
High86.3°–109.6°67201 / 4020.096 [0.047, 0.141]0.971 [0.804, 0.979]
All30.0°–109.6°200600 / 1,2000.093 [0.065, 0.121]0.968 [0.938, 0.984]

Across the sampled viewing-angle range, we observe no clear monotonic trend in either metric. This analysis characterizes estimated viewing-angle spread; capture distance, camera baseline, and view overlap are not separately controlled.

Revision. We will add the outdoor pilot and both breakdowns to the paper, and describe the UMGaze evidence as transfer to new indoor scenes and participants plus an initial outdoor measurement on two scenes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remark

Author Final Remark

We sincerely thank the reviewers and the Area Chair for the time and care invested in evaluating our submission. We appreciate the recognition of GazeWorld's key strengths: a new and practical problem setting that removes camera calibration and pre-associated head boxes, a complete contribution spanning the task, the framework, a large-scale training dataset, and an out-of-domain benchmark, and a controlled ablation showing that explicit geometry improves localization beyond pretrained features.

During the rebuttal period, we addressed every point raised in the reviews with new experiments:

  • Calibration-free 3D localization (rFPy). With predicted geometry and predicted view eligibility, GazeWorld is 0.22 m more accurate than the calibrated MVGT baseline on common frames, while covering more frames.
  • Independent random audit of EgoExo4D-Gaze (rFPy). Three blind raters judged 94.0% of released groups correct; we also report how filtering shifts the activity distribution.
  • Generalization (rFPy, Z6yu). UMGaze breakdowns by participant and camera configuration with confidence intervals, and a new outdoor pilot captured with a handheld phone, mostly at night.
  • Out-of-frame detection (rFPy). Precision and recall on head-invisible MVGT views and on EgoExo4D-Gaze.
  • Computational cost (Z6yu). Parameters, memory, and latency, compared with the specialized baselines.
  • Practical meaning of the 3D error and a human reference (sLm7). A median angular error of 9.4° at a median eye-to-target distance of 2.1 m, and a study with six human raters.
  • UMGaze (sLm7). What makes it out of domain, together with heatmap AUC and a 3D proxy metric.

Recruiting raters for the human-reference study and the independent annotation audit took additional time, so our responses were posted late in the discussion period. We would be grateful if they could be taken into account in the AC-reviewer discussion.

Revised paper

Changes are marked in blue, and the main text stays within the 10-page limit. The title is unchanged; one phrase of the abstract now describes UMGaze as handheld captures in new indoor scenes.

  • Main text. New Table 3 (calibration-free 3D localization); updated Table 2 (re-evaluated EgoExo4D-Gaze results, UMGaze AUC and 3D proxy); new Fig. 2(d) (outdoor example); short paragraphs on computational cost, the outdoor pilot, and the human reference; a clearer description of UMGaze in Section 5.
  • Appendix. F.2 (random audit and filtering bias), G.2-G.9 (the analyses listed above), an expanded Limitations section (C), and additional qualitative results with failure cases (G.10).

We will release the code, models, EgoExo4D-Gaze, and UMGaze. Thank you again for your thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.