acceptodds
Under review as a conference paper at ICLR 2027

PAGE: Towards Practical Human-level Gaze Target Estimation

Abstract

Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features through repeated cross attention. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code is available in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaze Target Estimation, ViT, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GpW4OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PaGE for third-person gaze target estimation. It combines scene–head interaction modules, spatially aligned positional encoding, staged backbone fine-tuning, and feature distillation on unlabeled images. The resulting models improve performance on three established gaze benchmarks, transfer to additional evaluation sets, and provide smaller students with substantially reduced computation.

Critical strengths
  1. The empirical improvements are substantial and cover multiple datasets and model sizes. The zero-shot results in Tab. 4 provide useful evidence beyond the datasets used for supervised training.

  2. The paper systematically develops its architecture and training recipe. Tab. 1 and the additional architectural and distillation ablations make the implementation choices unusually transparent.

  3. The lightweight students and the inference measurements increase practical value. The discussion of heatmap / coordinate bias is also useful.

Critical weaknesses
  1. The headline claim of human-level performance needs a narrower interpretation. On GazeFollow, agreement with a multi-annotator consensus is different from measuring gaze against independently observed eye-tracking ground truth. In addition, VAT and ChildPlay localization errors remain above the reported human baselines.

  2. The contribution of individual architectural changes is less certain than the overall system improvement. Please clarify which validation data governed design selection and provide uncertainty estimates for the central SIM versus token-concatenation comparison.

  3. Practical deployment is supported by desktop GPU/CPU profiling, but the broader robot and consumer-device claim remains less established.

Questions for authors
  1. How were the human baselines constructed, and how comparable are their inputs and annotation protocols to the model evaluations? Can the claims explicitly distinguish consensus prediction from physical gaze accuracy?

  2. Does the 16-ms GPU timing include head detection and preprocessing, and how does cost change with the number of queried people?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GpW4 (1/3)

Thanks for your support. We hope the results below address your concerns.

Q1 Human Baseline: The human baselines are the official results reported in the dataset papers. These numbers have since been cited extensively in gaze target estimation literature. The human baseline evaluation protocols for the three datasets are summarized below.

  • The GazeFollow paper [1] reports that each test set image is labeled by multiple annotators, and describes evaluating one annotation against the remaining ones when computing the human baseline.
  • The VAT paper [2] reports two additional annotators labeled test samples; human performance averages comparisons across annotator pairs.
  • The ChildPlay [3] human reference uses 2,000 double-coded instances, evaluating one annotation against the other.

L2 error, the localization metric, is directly comparable between the human baselines and our model. For the In/Out AP comparison, the model outputs a probability in the range [0, 1], while humans provide a binary classification (either 0 or 1). While its suitability is debatable, it has been used extensively in previous work and we inherit this setup. The comparability of AUC is less certain since humans annotate a point, not a heatmap, although many previous paper still include it in the comparison. We also agree that the human results are inter-annotator agreement metrics, not physical gaze accuracy computed with actual ground truth. Taking these factors into consideration, we will modify our main claim from "human-level performance" to "matching reported human inter-annotator gaze localization metrics (L2) on GazeFollow while significantly closing the gap on VAT and ChildPlay".

References

[1] Recasens, Adria, et al. "Where are they looking?." Advances in neural information processing systems 28 (2015).

[2] Chong, Eunji, et al. "Detecting attended visual targets in video." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[3] Tafasca, Samy, Anshul Gupta, and Jean-Marc Odobez. "Childplay: A new benchmark for understanding children’s gaze behaviour." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

AuthorsOpenReview ↗

Response to Reviewer GpW4 (2/3)

Q2 Deployment & Profiling: The 16-ms GPU timing includes head detection but not preprocessing. We ran the experiment on the RTX 4090 again to include preprocessing. Below are additional results with different number of heads.

Person Per FrameFPS on RTX 4090
146.23
243.63
441.52
836.75

For profiling on robots and mobile devices, we deployed PaGE ViT-S on an Nvidia Jetson AGX Orin 64GB, a common edge AI and robotics platform using the TensorRT backend, and an iPhone 17 Pro Max using the Core ML backend to gauge the model's edge performance. All results here use the standard 512*512 scenes and 256*256 heads, and include preprocessing, head detection, and PaGE inference.

Person Per FrameFPS on Jetson AGX Orin
116.45
215.02
411.65
88.01
Person Per FrameFPS on iPhone
131.55
224.57
417.03
89.96
AuthorsOpenReview ↗

Response to Reviewer GpW4 (3/3)

W2 Validation-set Results: During the development of PaGE, we used a held-out validation set that contained 10% of GazeFollow train set images chosen randomly, using the remaining 90% as the train set. Since GazeFollow provides 1 annotation per image for the train set instead of several annotations per image for the test set, there is no longer a distinction between Avg L2 and Min L2, and the results below are not directly comparable to GazeFollow test set results. Here are the results (mean$\pm$std over 3 random seeds) for key head branch design choices from our development logs.

StepAUCL2
Gaze-LLE w/ DINOv30.9627$\pm$0.00020.1152$\pm$0.0004
Token-concat Head Branch0.9648$\pm$0.00030.1108$\pm$0.0005
3 Layers →\rightarrow 5 Layers0.9652$\pm$0.00020.1086$\pm$0.0004
DINO Feature Dropout0.9658$\pm$0.00010.1085$\pm$0.0006
SIM Head Branch0.9664$\pm$0.00010.1074$\pm$0.0002
Reviewer_GpW4OpenReview ↗

Reply to authors

Thanks for your additional information, my concerns are solved. I'll keep my original score.

Reviewer_q1MJOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes PaGE, a head-box-conditioned gaze target estimation model built by incrementally modifying Gaze-LLE: replacing DINOv2 with DINOv3, training on a merged GazeFollow + VAT + ChildPlay set, adding a head-crop branch whose tokens interact with scene tokens through cross-attention (SIM), adding registers, GEGLU, 2D RoPE, and fine-tuning the backbones after the decoder has been trained. A ViT-H+ teacher is then distilled into ViT-S/S+/B students using 1.17M heads from unlabeled MPII and OpenImages images. The paper reports state-of-the-art results on GazeFollow, VAT and ChildPlay, claims to be "the first gaze estimation model with human-level performance", and adds experiments on screen gaze, a correction to the L2 evaluation, and qualitative results on animals, cameras and animation.

Critical strengths

1. Strong and consistent experimental results.

The method demonstrates strong and consistent performance across GazeFollow, VAT, and ChildPlay. Its results on the zero-shot GOO-Real benchmark further support its generalization capability.

2. Good reproducibility.

The authors provide the implementation code with the paper. I was able to run and validate the code successfully on my local machine, which is a valuable strength in terms of reproducibility.

3. Informative experimental analyses.

The step-by-step study, distillation analysis, and experiments with alternative backbones provide useful insights into the method's behavior and practical implementation. These analyses are particularly helpful for practitioners.

Critical weaknesses

1. The "human-level" claim, which is in the title and is the main claim of the paper (L21-23, L38-39, L229, L478), is a strong overclaim that is not supported by a like-for-like comparison.

I am particularly concerned about this point because it strongly weaken the motivation.

  • The "human performance" reported for GazeFollow, VAT and ChildPlay is an inter-annotator agreement measure, not human accuracy against true gaze, since these datasets have no ground-truth gaze. Exceeding it indicates that the model matches annotator consensus, not that it reaches human-level gaze perception.
  • The human baselines are taken from the original dataset papers (L144-145) and were computed under different protocols. AUC for a human is computed from a single click, whereas the model outputs a full heatmap, so "outperforming humans" in AUC (3 of the 7 metrics) does not compare the same quantity.
  • On GazeFollow, a model trained to regress the annotator distribution is evaluated against the mean of ~10 annotations, while the human baseline is one annotator against the others. Beating an individual annotator under this metric is expected from a consensus predictor, as the authors themselves note (L316-317), and does not mean the model is better at locating where the person actually looks.
  • On the only metric that is directly comparable on VAT and ChildPlay, the single-annotation L2, PaGE remains clearly worse than humans (0.064 vs 0.051, and 0.069 vs 0.048; Table 3).
  • The paper itself shows that the benchmark annotations contain errors, e.g. annotators labeling the target of a different person (App. K, Fig. 8). A model trained and evaluated on these labels can fit such annotator biases, so agreement with them cannot establish performance beyond humans.

2. The claim that the model utilizes eye orientation is insufficiently supported. (L402–403, L446–447)

The paper repeatedly emphasizes that the model can leverage eye orientation to improve gaze target estimation, but provides no direct evaluation of whether its predictions actually depend on eye information.

This capability is particularly important when eye orientation conflicts with head pose. Such cases exist in VAT, where humans can identify gaze targets based on eye direction even when the head is oriented elsewhere, while previous models such as Gaze-LLE may rely primarily on head pose.

I suggest evaluating these conflicting cases separately, comparing PaGE with previous methods, to demonstrate whether the proposed model genuinely uses eye orientation rather than predominantly relying on head pose.

3. The comparison with the baseline is not consistent and inflates the gains.

  • Table 1 re-evaluates Gaze-LLE with the "inout" checkpoints on GazeFollow (L142-144), i.e. a model fine-tuned on VAT, which gives 0.1058 Avg L2 for ViT-L. Table 3 instead uses the numbers reported in the Gaze-LLE paper (0.099, L311). The "road to PaGE" therefore starts from a weakened baseline on GazeFollow.
  • Parameter counts are not reported in any table, and FLOPs are reported only for PaGE (Table 2). The scene and head backbones are fine-tuned separately (L226-227, L267-268), so PaGE ViT-H+ has two 840M-parameter backbones, not "an 840M backbone" (L234). I trained and tested your model locally, the full param of PaGE ViT-H+ is 1.7B. Its comparison to Gaze-LLE ViT-L (one frozen backbone) should state parameters and FLOPs.

4. The "practical" claims are not substantiated.

  • The abstract and L41 claim real-time use "on robots and many consumer devices", but the only profiling is on an RTX 4090 (16 ms) and a desktop i9-14900K CPU (190 ms, ~5 FPS). No embedded or mobile device is evaluated.
  • PaGE requires a head detector and runs the head branch and the SIM decoder once per person, since the scene tokens are updated by cross-attention with each head. Cost as a function of the number of people, including the detector, is not reported, although multi-person scenes are the common case in the targeted applications.
  • L285-286 states "roughly 5%-10% of the teacher's compute", but Table 2 gives 4.1% (ViT-S) to 11.9% (ViT-B); the ViT-B student, which is the one called human-level in Table 3, uses 11.9%.

5. Novelty is very limited for ICLR.

The head branch follows earlier two-branch models in gaze target estimation, SIM is a cross-attention adapter (acknowledged as similar to ViT-Adapter, L807-808), registers, GEGLU and 2D RoPE are standard, the backbone is swapped for DINOv3, and freeze-then-fine-tune and feature distillation on unlabeled data are established techniques. The paper is a careful engineering study upon Gaze-LLE (CVPR 2025), but the conceptual contribution is small, and each individual architectural change improves GazeFollow Avg L2 by at most ~0.004, while the largest gains come from backbone fine-tuning and scaling.

6. All design decisions and hyperparameters are selected on the test sets.

  • No validation split is mentioned anywhere. Each step in Table 1 is kept or dropped based on test results, and the data mixture (Table 8, "yields the best overall result", L722-723), number of layers, token dimension, MLP ratio, FFN type, dropout, RoPE θ and learning rate (Tables 11-17) are all chosen on GazeFollow/VAT/ChildPlay test.
  • Many of these choices are within noise and inconsistent across metrics. For example, register tokens worsen GazeFollow Avg/Min L2 and VAT AP but are kept (Table 1), 2D RoPE worsens VAT and ChildPlay L2, and Table 11 varies AUC by only 0.0002 across depths. No seeds or variance are reported.

Minor Weaknesses but also important to address.

1. Several numbers and statements are inconsistent or unsupported.

  • Table 4: all three students outperform the ViT-H+ teacher on the Zhang et al. benchmark (72.4–76.3 vs 71.98). This is not discussed and suggests that the benchmark is too small or noisy to rank models.

2. Potential overlap between distillation data and evaluation sets.

COCO is excluded to avoid leakage (L246-247), but GazeFollow test images also come from other public image sources, some of which are Flickr-based like OpenImages. A near-duplicate check of the distillation set against all test sets should be reported.

3. Unsupported claims in the screen-gaze evaluation.

Sec. 4.3: the screen-gaze task is said to require "spatial perception and reasoning beyond human capabilities" (L387-388) without evidence. The evaluation uses only 3 users.

4. The discussion of animal gaze and visual attention is insufficiently justified.

The paper's claims about extending gaze target estimation to animal visual attention are questionable. Animals such as gazelles and penguins have different eye placements, fields of view, and visual perception compared with humans. These physiological differences make human-based gaze assumptions difficult to transfer directly to animals. More importantly, predicting a gaze point does not necessarily indicate what an animal can see or is attending to. The paper should distinguish gaze target estimation from visual attention estimation and avoid unsupported claims about animal perception.

Questions for authors

1. Overall Assessment.

This is a timely and important paper, but its contribution is undermined by overclaims and several inaccurate statements.

2. Response to the Main Concerns.

Each main argument of my review above is followed by supporting evidence. Please address each main argument and the evidence behind it.

3. Fair Comparison with Human Performance.

Please compute the human and model metrics under the same protocol (e.g., leave-one-annotator-out on GazeFollow for both, and point-based AUC for the model), or remove the human-level claim from the title and abstract.

4. Validation-Based Ablation Studies.

Please redo the main ablations (Table 1, Tables 8 and 11–17) on a validation split held out from training.

5. Computational Cost and Practical Deployment.

10 comments
AuthorsOpenReview ↗

Response to Reviewer q1MJ (1/6)

We are glad that you find the paper timely, important, and reproducible. We hope the response below addresses your concerns.

W1-1 Human-level Claim: We agree that the human baselines reported in the dataset papers are inter-annotator agreement metrics. However, we disagree that this "strongly weakens the motivation". The motivation behind our architectural and training recipe innovation is grounded in extensive experimental results. We will modify our main claim from "human-level performance" to "matching reported human inter-annotator gaze localization metrics (L2) on GazeFollow while significantly closing the gap on VAT and ChildPlay" to reflect the results more accurately.

W1-2 Dataset Annotation Errors: As for the occasional dataset annotation errors, we do not find them to be overly problematic after the main claim is modified.

W2 Eye Orientation: While VAT contains cases where head and eye orientations do not match, these cases are not annotated. Instead, Zhang et al.'s benchmark [1] explicitly explores this case and provided the necessary annotations. We report results on the subset (701 out of 2273 images) of Zhang et al.'s benchmark where the head and eye are oriented towards different objects. The results are consistent with Zhang et al.'s finding that MLLMs and previous specialist gaze models like Gaze-LLE overwhelmingly rely on head orientation, and perform worse than random guessing in cases where the head and eye orientations disagree [1]. PaGE, in contrast, performs significantly above chance, which supports our interpretation that the PaGE leverages eye orientation.

ModelCorrect / TotalAccuracy
Gaze-LLE ViT-L135 / 70119.26
ChanceN/A30.78
PaGE ViT-S Distill427 / 70160.91
PaGE ViT-S+ Distill436 / 70162.20
PaGE ViT-B Distill451 / 70164.34
PaGE ViT-H+ Teacher410 / 70158.49

Reference

[1] Zhang, Zory, et al. "Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

AuthorsOpenReview ↗

Response to Reviewer q1MJ (2/6)

W3-1 Baseline Fairness: We did not inflate the gains. We evaluate the Gaze-LLE In/Out baseline in Table 1 on GazeFollow because it is a step-by-step comparison and we used models trained with the In/Out prediction head for fairness. Table 3 provides a comparison of headline results reported in previous work. In addition, whether the baseline is 0.1058 or 0.099 does not change the main conclusions when PaGE ViT-L achieves a GazeFollow Avg L2 of 0.0823. The improvement remains significant. We have been fully transparent with which checkpoint we used throughout the paper, and we stand by the reported results.

W3-2 FLOPs and Parameter Count: We provide a FLOPs and parameter comparison with different number of input head boxes. Note that PaGE ViT-S Distill outperforms Gaze-LLE ViT-L on all benchmark metrics and use less compute even at 10 head boxes per image.

ModelParams (M)1 Head2 Heads3 Heads4 Heads5 Heads10 HeadsAdditional FLOPs per Head
Gaze-LLE ViT-B89.4222.8231.4240.0248.6257.2300.28.6
Gaze-LLE ViT-L307.3732.7741.3750.0758.5767.2810.28.6
PaGE ViT-S Distill59.096.9139.3181.6224.0266.4478.242.4
PaGE ViT-S+ Distill73.2115.2161.3207.3253.4299.4529.746.1
PaGE ViT-B Distill187.3283.1360.2437.3514.3591.4976.877.1
PaGE ViT-H+ Teacher1697.42373.62853.13332.73812.24291.86689.5479.5
AuthorsOpenReview ↗

Response to Reviewer q1MJ (3/6)

W4 Practical Deployment: We deployed PaGE ViT-S on Nvidia Jetson AGX Orin 64GB (a common platform for robotics and edge intelligence) and an iPhone 17 Pro Max to test PaGE's performance on actual robots and mobile devices. We use the TensorRT backend on Jetson and the Core ML backend on the iPhone. Here are the results (head detection and preprocessing included). The results indicate that PaGE can support fast inference on robots and mobile devices.

Person Per FrameFPS on Jetson AGX Orin
116.45
215.02
411.65
88.01
Person Per FrameFPS on iPhone
131.55
224.57
417.03
89.96

As for "roughly 5%-10% of the teacher's compute" corresponding to "4.1% to 11.9%", thanks for calculating the exact range, which we will put into the updated PDF.

Reviewer_jmCiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the PaGE, a gaze estimation model that explicitly captures the complex interplay between scene features and head features through repeated cross-attention. It integrates DINOv3-based encoders for both scene and head inputs, employing repeated cross-attention and positional encoding within a unified coordinate system. The training pipeline utilizes staged fine-tuning and token-level feature distillation, wherein a large PaGE teacher model transfers knowledge to smaller student models using images without gaze annotations.

Despite its strong performance, my primary concern is that the novelty may be limited. The proposed approach appears to be a combination of existing techniques, with performance gains largely stemming from the use of a more powerful pre-trained backbone, supplemented by engineering adjustments tailored specifically for this task.

Critical strengths

Experiments show that the proposed method achieves a noticeable performance improvement. Distillation effectively narrow the performance gap between model of different size.

Critical weaknesses
  1. The proposed method appears to be an incremental engineering extension of existing methods, offering limited novelty. The proposed PGE primarily integrates established techniques, such as cross-attention, staged fine-tuning, and feature distillation. To my knowledge, the technical report for DINOv3 already incorporates a distillation method.

  2. Although the proposed method outperforms human-level performance in 7 out of 9 metrics, it may not be entirely appropriate to claim that it surpasses human-level capabilities based solely on the number of metrics won. Furthermore, the inference speed of approximately 62.5 FPS mentioned in the paper was measured using an RTX 4090 GPU with a relatively low input resolution. It is recommended to include comparisons across a broader range of application scenarios.

Questions for authors
  1. Please elaborate on the core novelty of the proposed method. Methods with strong empirical performance but limited novelty may be better suited for workshop papers at CVPRW rather than full submissions to ICLR.
  2. For the ablation on dataset scale in distillation experiments, are the same number of training epochs used, meaning that the total number of updates increases with the dataset size? Furthermore, given the same dataset and a fixed training budget, what advantages does the proposed distillation method offer compared to simple output distillation?
  3. What annotator split and evaluation protocol are used to obtain the human reference results in Table 3? L2 errors on VAT and ChildPlay datasets are still higher than human baselines, how do the authors justify the human-level claim? It is recommended that the authors provide uncertainty analysis for these critical gaps, or clearly scope the set of capabilities to which this claim applies.
5 comments
AuthorsOpenReview ↗

Response to Reviewer jmCi (1/4)

We thank Reviewer jmCi for the insightful review and constructive suggestions. We hope the results below address your concerns.

Q1 Novelty: The novel contribution of the proposed method can be summarized as two points: (1) the finding that the usefulness of head information depends on how head and scene representations interact, which leads to a cross attention-based head branch for head-conditioned gaze decoding, and (2) a feature distillation recipe effectively leveraging unlabeled images that enables lightweight models to retain most of the teacher model's performance.

  • Attribution of Gains: Leveraging a more powerful backbone (DINOv3) is not the main driver of PaGE's performance. As shown in Table 1 of the paper, the largest gains come from the head branch and SFT, not switching from DINOv2 to DINOv3. Table 9 in Appendix C further demonstrates that a PaGE ViT-L model with DINOv2 still significantly outperforms Gaze-LLE ViT-L even without SFT (e.g. 0.0856 VAT L2 vs. Gaze-LLE ViT-L's 0.1035).
  • Head Branch: A key finding of Ryan et al. in Gaze-LLE [1] is that a head branch brings negligible added benefit to a ViT gaze decoder working on top of a pre-trained general purpose encoder. Our proposed SIM module demonstrates that, contrary to previous findings, a head branch does bring significant benefits when executed properly. Our progression from no head branch, to the token-concat head branch, and then to SIM provides detailed documentation of how we reworked the head branch back into gaze estimation model and the motivation behind each modification.
  • Distillation: Distilling gaze estimation models is not a trivial application of knowledge distillation, as simple output distillation is much less effective on gaze data (see our response to Q2 below for details). Our distillation recipe is distinct from output distillation and DINO-style distillation (which distills DINO head prototype predictions) [2], and can be readily reused in future gaze estimation work.

References

[1] Ryan, Fiona, et al. "Gaze-lle: Gaze target estimation via large-scale learned encoders." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." 2021 IEEE/CVF international conference on computer vision (ICCV). IEEE, 2021.

AuthorsOpenReview ↗

Response to Reviewer jmCi (2/4)

Q2 Distillation: The gains from a larger, more diverse distillation dataset persist when we match the total number of training steps. We also find that the proposed feature distillation method outperforms simple output distillation, in addition to providing a workaround to the lack of ground truth in/out labels in distillation data.

  • Matched-compute Distillation Experiment: We compare the performance of PaGE ViT-B students distilled with 200k, 400k, and 1.17M samples. The total number of training steps match that of the "20 epochs over 1.17M samples" regime used in the paper. All students underwent the same SFT process described in the paper. While additional compute does improve performance, the larger, more diverse 1.17M distillation set still brings the best overall performance.

    GazeFollowVATChildPlay
    AUCAvg L2Min L2AUCL2APAUCL2AP
    200k, 20 epochs0.96470.08390.03160.96580.07180.94840.97100.07280.9958
    400k, 20 epochs0.96510.08340.03070.96660.07210.94150.97130.07190.9965
    200k, matched compute0.96580.08180.03010.96840.06900.94580.97240.07290.9960
    400k, matched compute0.96600.08090.02970.96740.06850.94290.97290.07050.9971
    1.17M, matched compute0.96600.08140.02950.96880.06770.94500.97340.06970.9969
  • Feature Distillation vs. Output Distillation: A problem with output distillation for gaze estimation is that the unlabeled samples lack an in/out label, which decides whether the heatmap loss for that sample is used to update the model during training. This is also a motivation behind our feature distillation method, which circumvents the problem entirely.

    We compare feature distillation with two variants of output distillation, one where the heatmap loss of all samples is used to update the student, and one where the heatmap loss of samples labeled by the teacher as "in-frame" is used to update the student. All experiments here are done with 200k samples over 20 epochs. Output distillation uses the same loss as supervised training, and all students underwent the same SFT process described in the paper. Feature distillation outperforms both output distillation variants.

    GazeFollowVATChildPlay
    AUCAvg L2Min L2AUCL2APAUCL2AP
    200k, 20 epochs, output distillation0.96310.08890.03420.95990.07900.94840.96740.07860.9961
    200k, 20 epochs, output distillation (teacher "in-frame" samples for heatmap loss)0.96240.08990.03520.95980.07980.93990.96670.07870.9945
    200k, 20 epochs0.96470.08390.03160.96580.07180.94840.97100.07280.9958
AuthorsOpenReview ↗

Response to Reviewer jmCi (3/4)

Q3 Human Baseline: The human baselines are the official results reported in the dataset papers. These numbers have since been cited extensively in gaze target estimation literature. The human baseline evaluation protocols for the three datasets are summarized below.

  • The GazeFollow paper [1] reports that each test set image is labeled by multiple annotators, and describes evaluating one annotation against the remaining ones when computing the human baseline.
  • The VAT paper [2] reports two additional annotators labeled test samples; human performance averages comparisons across annotator pairs.
  • The ChildPlay [3] human reference uses 2,000 double-coded instances, evaluating one annotation against the other.

Since the annotations used for the VAT and ChildPlay human baselines are not publicly released, it is difficult to conduct a proper uncertainty estimate. We agree that broad equivalence with human gaze following performance is hard to rigorously establish without significant additional experiments involving real humans. We will modify our main claim from "human-level performance" to "matching reported human inter-annotator gaze localization metrics (L2) on GazeFollow while significantly closing the gap on VAT and ChildPlay". We will also explicitly discuss the "human baseline" numbers as inter-annotator agreement metrics and the associated limitations of the comparison. We will update the PDF before the end of the discussion period.

References

[1] Recasens, Adria, et al. "Where are they looking?." Advances in neural information processing systems 28 (2015).

[2] Chong, Eunji, et al. "Detecting attended visual targets in video." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[3] Tafasca, Samy, Anshul Gupta, and Jean-Marc Odobez. "Childplay: A new benchmark for understanding children’s gaze behaviour." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

Comments on the paper

1 comment

Dear Reviewers,

Thanks for providing detailed feedback on our paper. We have posted the responses to your reviews, and would gladly engage in further discussion. We will post the revised PDF later to include other changes that could potentially arise during discussion.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.