acceptodds
Withdrawn from ICLR 2027

LookAnything: A Generalist Foundation Model for Gaze Target Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human gaze provides a rich window into attention and intent, making its understanding essential for applications ranging from social signal analysis to human-robot interaction. Existing methods typically employ highly customized architectures relying on auxiliary inputs like head crops or depth maps, causing cascading errors and inference overhead. Trained on single domains and confined to isolated gaze target detection with heatmap outputs, they lack flexibility and generalization for diverse interactive scenarios. To bridge this gap, we present LookAnything, a generalist foundation model unifying diverse gaze-centric tasks. Benefiting from large-scale multi-task cross-domain training, LookAnything flexibly adapts to various settings, generating diverse outputs (points, boxes, heatmaps, and language) and supporting broad gaze-centric localization, description, and reasoning. At its core, we introduce learnable gaze-prior queries to promote gaze-aware representation learning, eliminating the need for auxiliary external inputs. To facilitate this paradigm, we construct HoliGaze, a comprehensive corpus and benchmark comprising 311K question-answer pairs across 5 capability dimensions and 9 distinct tasks. Experiments demonstrate that LookAnything consistently outperforms 33 leading vision-language models on holistic gaze understanding, while remaining highly competitive with dedicated specialists across 5 conventional target detection benchmarks. Deployment on a Unitree G1 robot further validates its strong out-of-domain generalization and flexible adaptation in real-world scenarios. Our codes, model, and dataset will be released. The project website is available at https://anonymous1541paper.github.io/iclr2027-1541.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaze Target Detection, Attention Modeling, Human-Robot Interaction, Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LJqJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LookAnything, a gaze model built on Qwen3-VL-2B. The model introduces three sets of learnable "gaze-prior" queries guided by auxiliary decoders, decoding points, boxes, and text autoregressively. To train the model, the authors construct HoliGaze, a 311K QA corpus and benchmark (note most of images are from existing datasets) across 9 templated tasks. Missing annotations, such as referring expressions, head orientations, depth, head masks, and multiple-choice distractors are filled in using Qwen3-VL-32B, Qwen3-32B, SAM3, and Depth Anything 3. LookAnything outperforms zero-shot VLMs on HoliGaze-Bench, rivals specialist models across 5 standard gaze-following benchmarks, and demonstrates practical feasibility through qualitative experiments on a robot.

Critical strengths
  • Unifying localization, description and refusal-style gaze tasks into a single interface is an interesting direction, and the task taxonomy is clearly presented.
  • The data engine is documented in detail. I like this part.
  • Injecting geometric priors into a VLM through learnable queries with dense auxiliary supervision is a sensible design. And I remember some old gaze modeling papers did this before but seldom of them try it with VLMs.
Critical weaknesses
  • The central claim of "eliminating auxiliary inputs" (L21-22, L93, L478) is not supported by the formulation.

    • The model still needs to be told who the source person is. On HoliGaze, this is a text expression generated by Qwen3-VL-32B from the GT head box (App. D.2). In GTL-B2P, it is the head box itself (Table 10). At deployment, the text reference has to come from somewhere, which reintroduces exactly the external module the paper claims to remove.
    • Head information is moved from the input to the supervision, not eliminated.
    • L14-15, L43-44 and L247-248 state that head/depth inputs "introduce cascading errors and latency". No evidence is provided for either. Recent models like PaGE proves that cropped head has strong benefit for this task.
    • Unlike the short-expression prompt (Fig. 18), the long-expression prompt (Fig. 16) does not forbid head-orientation words, and the generated references do contain them: "in side profile" (L109), "seen in side profile" (Fig. 28), "seen from behind" (Fig. 33). The text prompt therefore leaks gaze-relevant cues that box-prompted baselines do not receive.
  • The evaluation protocol on native benchmarks (Table 3) is unspecified and partially contradicts the method description.

    • The paper never states how the subject is specified for Table 3, nor which output produces the L2.
    • Fig. 1 labels the auxiliary branches "training-only", Fig. 4 labels them "Inference-Free", and L242/L827-828 state they can be skipped. However, L318 states that AUC is computed from the query-based heatmap, i.e. the auxiliary decoder. Either the decoders are required at inference, or L2 and AUC come from different heads. In both cases, the "single language interface" claim does not hold for Table 3.
    • In/out-of-frame prediction is not evaluated. VAT and ChildPlay are standardly evaluated with AP_in/out, and Table 1 itself counts 2 tasks for these datasets. The only related task, PV, is about 200 samples of HoliGaze-Bench and consists of emitting a fixed refusal string. This does not replace the standard protocol.
    • LookAnything is trained jointly on the training splits of all 5 benchmarks plus additional data (Table 9), while most specialists are trained per dataset. The source of the baseline numbers on GazeHOI and GOO-Real is not given (retrained by the authors? with which settings?). The "Avg." of Tafasca et al. (2023) is computed over 3 datasets only and is not comparable.
    • L399-400 "matches or exceeds dedicated specialists": on GazeFollow, LookAnything is worse than Gaze-LLE on both Min L2 (0.048 vs 0.041) and AUC (0.945 vs 0.958). And it will be even much worse than PaGE (https://arxiv.org/abs/2607.04860). The author mentioned PaGE in their reference but never have a comparison.
  • HoliGaze-Bench mostly measures agreement with the authors' own generation pipeline, and the 33-VLM comparison is not informative.

    • Referring expressions and GTD/GSD reference answers are generated by Qwen3-VL-32B, distractors by Qwen3-32B/SAM3, and GDO/GDR labels by Qwen3-VL-32B head orientation combined with DA3 depth (App. D.2). The model is trained on 303K samples from the same pipeline and templates, and its depth query is directly supervised with DA3. Scoring openQA with ROUGE-L/BLEU-4 against Qwen-generated references rewards matching that style.
    • All 33 baselines are evaluated zero-shot (L756). Beating zero-shot models on an in-distribution, templated benchmark after fine-tuning on it is expected and says nothing about the proposed design. The output format is not neutral either: the 0-1000 integer (x,y) convention is native to Qwen3-VL but not to other families, and parsing/failure handling for baselines is not described.
    • The MCQ distractors have exploitable structure: permutation and perturbation produce near-duplicate options sharing the correct box or description (e.g. Fig. 2 GSD-MCQ, Fig. 29, Fig. 33). A model fine-tuned on the same generator can learn these artifacts. An option-only/blind baseline is needed.
    • The 0.70% modification rate (L160-161, L1223-1226) is presented as evidence of quality, but a low edit rate on machine drafts is equally consistent with annotator anchoring. No inter-annotator agreement is reported, and the 303K training corpus is not verified at all.
  • The key baseline is missing: Qwen3-VL-2B fine-tuned on HoliGaze-Corpus without gaze-prior queries.

    • The headline gain (GTL-T2P 0.387 → 0.094, L368) compares the zero-shot backbone with the fully fine-tuned model, which mixes the effect of the data with the effect of the architecture. Fig. 7 only removes one query group at a time.
    • Where ablations are reported, the effects are marginal (native Avg. L2 0.109 vs 0.111/0.112, Tables 4-5), from a single run without variance. This suggests most of the gain comes from supervised fine-tuning on the data, not from the proposed queries. Comparisons with other gaze-tuned VLMs (https://arxiv.org/pdf/2511.06348, https://arxiv.org/pdf/2512.20735) are also absent.
  • No efficiency analysis, although efficiency is part of the motivation. LookAnything is a 2B autoregressive VLM, and its AUC requires an additional decoder. Latency, FLOPs and memory should be compared with Gaze-LLE-type, GazeAnywhere-type specialists.

  • Overclaimed contributions. "Generalist foundation model" (title, L72) and "the first generalist foundation model for holistic gaze understanding" (L724) for a 2B model fine-tuned for 2 epochs on 303K templated samples, whose test sources are the same as its training sources (no domain gaps). The only OOD evidence is qualitative robot footage (Fig. 8) with no quantitative metric, and there is no held-out-dataset evaluation. The author of the paper seems don't know the definition of the foundation model.

Some minor issues:

  • Several numbers in the text do not match the tables or the appendix:
    • L361 "0.152 vs 0.321 GTL-T2B mIoU" → Table 2 reports 0.333.
    • L451 "from 0.1118 to 0.1410" → Table 6 reports 0.109 and 0.138.
    • L323 peak LR 2e-5 → App. B.4 and Table 8 report 1e-5 (backbone) and 1e-4 (heads).
    • The task proportions in L199-200 do not match the pie chart in Fig. 3.
    • L887 "Following the main text, the loss weights..." → the main text gives no loss weights. Footnote 1 (L1240) acknowledges that the main-text description of distractor expansion is inaccurate for GOO.
  • Incorrect citations (I only check one paper I recently read...):
    • VL4Gaze is cited as "Wang et al., arXiv:2605.22607, 2026a" (L646-647), whereas the paper is arXiv:2512.20735 (2025), which is also what Table 1 ("arXiv 2025") implies.
  • App. E.1 is not a scaling analysis: it evaluates intermediate checkpoints of a single cosine-schedule run (0.25-2 epochs). This conflates training progress with data scale and does not support the "neural scaling laws" claim (L1275).
  • The ablation explanations ("gradient noise", "attention dilution", "representational purity", L419-427) are asserted without evidence. Fig. 7 uses tightly truncated y-axes, which visually exaggerate differences of ~0.002 L2.
  • The authors should tone down the language:
    • L308 "elegantly integrating them" → the paper gives no evidence for this.
    • L1224-1226 "exceptionally low rate ... guarantees a highly reliable evaluation set" → see the comment on anchoring above.
    • L95 "the first comprehensive corpus" → HoliGaze is a re-annotation of existing datasets.
  • Robot experiments: the Ethics Statement says "controlled laboratory environments" (L503), while Sec. 4.5 describes "shopping scenarios".
    • Participant consent is not mentioned. This is not a big ethic issue, but the author need to shortly response it here.
Questions for authors
  • The overall direction of the paper is not solid and not interesting to the gaze modeling community, thus I recommend strong reject of this paper. There are many issues in the paper that need to be resolved during rebuttal. My final rating will be changed a little if most of the problems can be resolved.
  • I have separated each of my main arguments into evidence. The author only needs to answer the main arguments (6 major concerns) in short and think about if the answer of each concern covered the explanation of most of the evidence. You don't need to write answer for each evidence because you can directly edit related sections in your paper. (I remember ICLR can let authors to edit paper last year in the rebuttal phase)
  • Specify the exact protocol for Table 3 (how the subject is specified, which head produces L2 and AUC), report AP_in/out on VAT and ChildPlay, and report results using the GT head box as the only subject specification.
  • Add a plain SFT baseline (Qwen3-VL-2B on HoliGaze-Corpus without queries), an option-only MCQ baseline, and latency/FLOPs against Gaze-LLE.
  • If there is no space in rebuttal windows, all minor issues can be skipped.
Reviewer_5JEAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents LookAnything, a multimodal model for "generalist" gaze understanding. It introduces gaze-prior queries with auxiliary depth, head-mask, and gaze-target supervision, and constructs HoliGaze, a derived dataset of 311K QA pairs covering 9 gaze-related tasks. The model is evaluated against general-purpose VLMs on HoliGaze-Bench, specialist gaze detectors on five established benchmarks, and qualitatively in a robot setting.

Critical strengths

S1) The learnable gaze-prior queries, together with auxiliary supervision from depth, head masks, and gaze-target heatmaps, form a coherent architectural design.

S2) The ablation studies suggest that these components are useful and contribute to the final performance. In particular, the results indicate that the different geometric priors have complementary roles. This is kind of predictable but interesting.

S3) The paper involves a significant amount of implementation and experimental effort. For example, evaluating the model across several benchmarks is non-trivial.

S4) Achieving strong performance with a relatively small 2B backbone is valuable, especially for real-time or resource-constrained deployment. However, the paper should distinguish this practical advantage from the claim of "fundamental superiority" over larger models.

Critical weaknesses

Before listing the weaknesses, I would like to clarify that the available soundness questions and rating options do not fully capture my assessment of this paper. I consider the paper technically sound overall, but substantial improvements are needed, as discussed below. Some components and experiments appear to have been added mainly to increase the perceived scope of the work, without clearly strengthening its central contribution. As a result, the paper sometimes feels overextended and makes it difficult for the reader to identify the main message and most important contributions. More experiments do not necessarily make a paper more convincing unless they directly support its core claims.

W1) The novelty is mainly an integration of several existing ingredients: VLM fine-tuning, spatial auxiliary supervision, learnable queries, and multi-task instruction tuning. This combination is potentially useful, but the paper does not yet establish that the proposed architectural mechanism itself is responsible for the improvements, rather than simply the addition of 303K gaze-derived training examples to Qwen3-VL. The paper does not clearly disentangle three possible sources of performance gains: (i) the use of more gaze-specific training data; (ii) multi-task instruction tuning; (iii) the proposed gaze-prior-query architecture. Several additional experiments would therefore be needed to support the paper’s strongest claims. At minimum, the authors should consider: (a) fine-tuning Qwen3-VL-2B on the same HoliGaze data without gaze-prior queries; (b) training the same architecture only on the conventional gaze-target detection task; (c) comparing the proposed gaze-specific queries with ordinary learned tokens; (d) evaluating the same auxiliary losses without the block-wise attention mechanism; These are not minor additional experiments. They are necessary to determine whether the main contribution is the architecture, the multi-task formulation, or simply the scale and composition of the training data. Without such controls, the claim that the queries “internalize gaze priors” remains plausible but not fully demonstrated.

W2) The paper presents previous gaze-target detectors as "inflexible" because they generally produce heatmaps and may require auxiliary inputs such as head crops or depth maps. This argument is reasonable if the intended application requires language-based reasoning, source identification, or multiple output formats as in this paper. However, heatmaps are not inherently inadequate for gaze estimation; they are often an appropriate representation, particularly given the uncertainty and subjectivity of gaze-target annotations. Similarly, explicit head or depth representations are not necessarily disadvantages. They may improve reliability, interpretability, or data efficiency. The paper would therefore benefit from framing its contribution as an expansion of the task scope and output interface, rather than suggesting that previous specialist methods are generally obsolete or fundamentally inadequate.

W3) The comparison in Tab. 2 is difficult to interpret as a fair comparison of gaze methods. The models included in that table are mostly general-purpose or embodied multimodal models, whereas LookAnything is specifically trained on HoliGaze and receives gaze-specific auxiliary supervision. The table therefore mainly demonstrates that generic models do not perform particularly well on this newly constructed gaze benchmark, while a gaze-specialized model does. But this result is not unexpected and does not by itself establish superiority over competing gaze-understanding methods. Thus, Table 2 should be presented explicitly as an analysis of how well general-purpose models transfer to gaze understanding. The authors should avoid presenting these models as equivalent competitors unless they also include recent gaze-tuned MLLMs, such as VL4Gaze, GazeVLM, OmniGF, and other relevant gaze-language approaches.

W4) In Table 3, it is not necessary to include every gaze paper in a single table, but the paper should state a clear and defensible criterion for selecting the baselines. The appendix currently claims that the methods represent the main architectural families, but this justification is insufficient when Table 3 is used to support broad state-of-the-art claims. For example, the omission of Object-aware Gaze Target Detection (ICCV) is difficult to justify. It is directly concerned with gaze-target detection, incorporates object-level target reasoning, and reports results on standard in-the-wild benchmarks. Its omission is especially noticeable because the paper itself motivates LookAnything partly through the need for instance-level and semantic gaze understanding.

W5) HoliGaze is not a newly collected gaze dataset in the conventional sense. It is a large-scale transformation of eight existing datasets into a unified set of questions, answers, descriptions, and task formats. This is still a valuable contribution, but the paper should describe it as a derived multi-task benchmark rather than as a new collection of 311K independently annotated gaze observations. The claim that only 56 out of 8,000 benchmark samples required modification does not demonstrate that the underlying gaze labels are reliable. It shows that the generated questions, descriptions, distractors, and geometric representations were accepted by the annotators. However, the central scientific uncertainty whether the annotated point or object genuinely corresponds to what the person is looking at is inherited from the source datasets and is not resolved by this verification process. This is particularly important given the subjectivity and ambiguity of gaze-target annotation. The AL-GTD paper is relevant here because it highlights the cost of obtaining large amounts of gaze-target supervision and investigates how competitive performance can be achieved with substantially less labeled data. I would discuss AL-GTD in relation to the necessity and cost of constructing such a large derived corpus. (a)The paper states that five annotators cross-checked every HoliGaze-Bench sample, but it is unclear whether they independently assessed the actual gaze target or only verified the automatically generated questions, descriptions, distractors, and geometric annotations. The authors should clarify the scope of this human verification and report inter-annotator agreement if independent gaze-target judgments were collected. If no such judgments were collected, the paper should avoid presenting the 0.70% modification rate as evidence that the underlying gaze annotations are reliable. (b) The paper does not report performance broken down by factors such as small or occluded targets, out-of-frame targets, social gaze, or human-object interactions. Such analyses would help determine whether the reported improvements reflect better gaze understanding or mainly easier cases and improved general visual-language reasoning.

W6) There is also a conceptual issue with the proposed task taxonomy. Gaze direction, depth ordering, contact detection, ambiguity recognition, and presence verification are not necessarily equivalent forms of gaze understanding. Some of these tasks may instead measure general visual reasoning, depth estimation, referring-expression comprehension, or refusal behavior conditioned on a gaze-related annotation. For example, a model might correctly answer whether a target is left and closer without learning a rich gaze representation. The authors should therefore justify the taxonomy more carefully and demonstrate that the benchmark measures gaze-specific competence rather than a collection of general vision-language abilities.

W7) There is a conceptual issue with the proposed task taxonomy. Gaze direction, depth ordering, contact detection, ambiguity recognition, and presence verification are not necessarily equivalent forms of gaze understanding. Some of these tasks may instead measure general visual reasoning, depth estimation, referring-expression comprehension, or refusal behavior conditioned on a gaze-related annotation. For example, a model might correctly answer whether a target is left and closer without learning a rich gaze representation. The authors should therefore justify the taxonomy more carefully and demonstrate that the benchmark measures gaze-specific competence rather than a collection of general vision-language abilities.

W8) The real-world robot experiment provides only qualitative evidence. A small number of example images with predicted boxes and heatmaps is not sufficient to support a strong claim about out-of-domain generalization or robustness. The experiment would be more convincing if the authors reported the number of scenes and participants, defined the evaluation protocol, included failure cases, and provided quantitative or structured human evaluation. Otherwise, it should be presented as a qualitative demonstration rather than as validation of robust generalization.

Questions for authors

The weaknesses list a lot of questions, but some major ones are reported below again.

Q1) Can the authors disentangle the gains from HoliGaze scale and multi-task training from the proposed gaze-prior-query architecture?

Q2) For Table 3, which results were reproduced by the authors and which were taken from prior work? What exact training data, preprocessing, and hyperparameter-selection protocol was used for LookAnything?

Q3) Did the five annotators independently assess the actual gaze target, or did they only verify the generated descriptions, boxes, distractors, and depth labels?

Reviewer_puerOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce LookAnything, a gaze-centric foundation model, and HoliGaze, a benchmark dataset for training and evaluating gaze-centric tasks. LookAnything is a generalist multimodal LLM-based model which undergoes large-scale multi-task cross-domain training to flexibly generalize to several task settings and scenarios. HoliGaze, on the other hand, is the first corpus and benchmark of its kind containing 311K Q&A pairs across 5 capability dimensions and 9 distinct tasks. Experimental evaluation shows that on HoliGaze, LookAnything outperforms all 33 best-performing VLMs on HoliGaze and remains highly competitive with specialist gaze-centric models.

Critical strengths
  1. The paper addresses a crucial gap in gaze-centric models - the absence of generalist foundation models supporting gaze as a modality, making it an important contribution in the field.

  2. The dataset and model design is carefully thought out and executed. The details are well-recorded and exhaustively detailed across the main text and appendix.

  3. The authors perform an exhaustive evaluation of a large number of baseline VLMs and specialist models along with their proposed model. The proposed model registers significant improvements in performance over the baselines.

Critical weaknesses

None

Questions for authors
  1. Did the authors consider any other backbone in addition to Qwen3-VL-2B? Does performance increase with higher number of parameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.