- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces LookAnything, a gaze model built on Qwen3-VL-2B. The model introduces three sets of learnable "gaze-prior" queries guided by auxiliary decoders, decoding points, boxes, and text autoregressively. To train the model, the authors construct HoliGaze, a 311K QA corpus and benchmark (note most of images are from existing datasets) across 9 templated tasks. Missing annotations, such as referring expressions, head orientations, depth, head masks, and multiple-choice distractors are filled in using Qwen3-VL-32B, Qwen3-32B, SAM3, and Depth Anything 3. LookAnything outperforms zero-shot VLMs on HoliGaze-Bench, rivals specialist models across 5 standard gaze-following benchmarks, and demonstrates practical feasibility through qualitative experiments on a robot.
Critical strengths
- Unifying localization, description and refusal-style gaze tasks into a single interface is an interesting direction, and the task taxonomy is clearly presented.
- The data engine is documented in detail. I like this part.
- Injecting geometric priors into a VLM through learnable queries with dense auxiliary supervision is a sensible design. And I remember some old gaze modeling papers did this before but seldom of them try it with VLMs.
Critical weaknesses
-
The central claim of "eliminating auxiliary inputs" (L21-22, L93, L478) is not supported by the formulation.
- The model still needs to be told who the source person is. On HoliGaze, this is a text expression generated by Qwen3-VL-32B from the GT head box (App. D.2). In GTL-B2P, it is the head box itself (Table 10). At deployment, the text reference has to come from somewhere, which reintroduces exactly the external module the paper claims to remove.
- Head information is moved from the input to the supervision, not eliminated.
- L14-15, L43-44 and L247-248 state that head/depth inputs "introduce cascading errors and latency". No evidence is provided for either. Recent models like PaGE proves that cropped head has strong benefit for this task.
- Unlike the short-expression prompt (Fig. 18), the long-expression prompt (Fig. 16) does not forbid head-orientation words, and the generated references do contain them: "in side profile" (L109), "seen in side profile" (Fig. 28), "seen from behind" (Fig. 33). The text prompt therefore leaks gaze-relevant cues that box-prompted baselines do not receive.
-
The evaluation protocol on native benchmarks (Table 3) is unspecified and partially contradicts the method description.
- The paper never states how the subject is specified for Table 3, nor which output produces the L2.
- Fig. 1 labels the auxiliary branches "training-only", Fig. 4 labels them "Inference-Free", and L242/L827-828 state they can be skipped. However, L318 states that AUC is computed from the query-based heatmap, i.e. the auxiliary decoder. Either the decoders are required at inference, or L2 and AUC come from different heads. In both cases, the "single language interface" claim does not hold for Table 3.
- In/out-of-frame prediction is not evaluated. VAT and ChildPlay are standardly evaluated with AP_in/out, and Table 1 itself counts 2 tasks for these datasets. The only related task, PV, is about 200 samples of HoliGaze-Bench and consists of emitting a fixed refusal string. This does not replace the standard protocol.
- LookAnything is trained jointly on the training splits of all 5 benchmarks plus additional data (Table 9), while most specialists are trained per dataset. The source of the baseline numbers on GazeHOI and GOO-Real is not given (retrained by the authors? with which settings?). The "Avg." of Tafasca et al. (2023) is computed over 3 datasets only and is not comparable.
- L399-400 "matches or exceeds dedicated specialists": on GazeFollow, LookAnything is worse than Gaze-LLE on both Min L2 (0.048 vs 0.041) and AUC (0.945 vs 0.958). And it will be even much worse than PaGE (https://arxiv.org/abs/2607.04860). The author mentioned PaGE in their reference but never have a comparison.
-
HoliGaze-Bench mostly measures agreement with the authors' own generation pipeline, and the 33-VLM comparison is not informative.
- Referring expressions and GTD/GSD reference answers are generated by Qwen3-VL-32B, distractors by Qwen3-32B/SAM3, and GDO/GDR labels by Qwen3-VL-32B head orientation combined with DA3 depth (App. D.2). The model is trained on 303K samples from the same pipeline and templates, and its depth query is directly supervised with DA3. Scoring openQA with ROUGE-L/BLEU-4 against Qwen-generated references rewards matching that style.
- All 33 baselines are evaluated zero-shot (L756). Beating zero-shot models on an in-distribution, templated benchmark after fine-tuning on it is expected and says nothing about the proposed design. The output format is not neutral either: the 0-1000 integer (x,y) convention is native to Qwen3-VL but not to other families, and parsing/failure handling for baselines is not described.
- The MCQ distractors have exploitable structure: permutation and perturbation produce near-duplicate options sharing the correct box or description (e.g. Fig. 2 GSD-MCQ, Fig. 29, Fig. 33). A model fine-tuned on the same generator can learn these artifacts. An option-only/blind baseline is needed.
- The 0.70% modification rate (L160-161, L1223-1226) is presented as evidence of quality, but a low edit rate on machine drafts is equally consistent with annotator anchoring. No inter-annotator agreement is reported, and the 303K training corpus is not verified at all.
-
The key baseline is missing: Qwen3-VL-2B fine-tuned on HoliGaze-Corpus without gaze-prior queries.
- The headline gain (GTL-T2P 0.387 → 0.094, L368) compares the zero-shot backbone with the fully fine-tuned model, which mixes the effect of the data with the effect of the architecture. Fig. 7 only removes one query group at a time.
- Where ablations are reported, the effects are marginal (native Avg. L2 0.109 vs 0.111/0.112, Tables 4-5), from a single run without variance. This suggests most of the gain comes from supervised fine-tuning on the data, not from the proposed queries. Comparisons with other gaze-tuned VLMs (https://arxiv.org/pdf/2511.06348, https://arxiv.org/pdf/2512.20735) are also absent.
-
No efficiency analysis, although efficiency is part of the motivation. LookAnything is a 2B autoregressive VLM, and its AUC requires an additional decoder. Latency, FLOPs and memory should be compared with Gaze-LLE-type, GazeAnywhere-type specialists.
-
Overclaimed contributions. "Generalist foundation model" (title, L72) and "the first generalist foundation model for holistic gaze understanding" (L724) for a 2B model fine-tuned for 2 epochs on 303K templated samples, whose test sources are the same as its training sources (no domain gaps). The only OOD evidence is qualitative robot footage (Fig. 8) with no quantitative metric, and there is no held-out-dataset evaluation. The author of the paper seems don't know the definition of the foundation model.
Some minor issues:
- Several numbers in the text do not match the tables or the appendix:
- L361 "0.152 vs 0.321 GTL-T2B mIoU" → Table 2 reports 0.333.
- L451 "from 0.1118 to 0.1410" → Table 6 reports 0.109 and 0.138.
- L323 peak LR 2e-5 → App. B.4 and Table 8 report 1e-5 (backbone) and 1e-4 (heads).
- The task proportions in L199-200 do not match the pie chart in Fig. 3.
- L887 "Following the main text, the loss weights..." → the main text gives no loss weights. Footnote 1 (L1240) acknowledges that the main-text description of distractor expansion is inaccurate for GOO.
- Incorrect citations (I only check one paper I recently read...):
- VL4Gaze is cited as "Wang et al., arXiv:2605.22607, 2026a" (L646-647), whereas the paper is arXiv:2512.20735 (2025), which is also what Table 1 ("arXiv 2025") implies.
- App. E.1 is not a scaling analysis: it evaluates intermediate checkpoints of a single cosine-schedule run (0.25-2 epochs). This conflates training progress with data scale and does not support the "neural scaling laws" claim (L1275).
- The ablation explanations ("gradient noise", "attention dilution", "representational purity", L419-427) are asserted without evidence. Fig. 7 uses tightly truncated y-axes, which visually exaggerate differences of ~0.002 L2.
- The authors should tone down the language:
- L308 "elegantly integrating them" → the paper gives no evidence for this.
- L1224-1226 "exceptionally low rate ... guarantees a highly reliable evaluation set" → see the comment on anchoring above.
- L95 "the first comprehensive corpus" → HoliGaze is a re-annotation of existing datasets.
- Robot experiments: the Ethics Statement says "controlled laboratory environments" (L503), while Sec. 4.5 describes "shopping scenarios".
- Participant consent is not mentioned. This is not a big ethic issue, but the author need to shortly response it here.
Questions for authors
- The overall direction of the paper is not solid and not interesting to the gaze modeling community, thus I recommend strong reject of this paper. There are many issues in the paper that need to be resolved during rebuttal. My final rating will be changed a little if most of the problems can be resolved.
- I have separated each of my main arguments into evidence. The author only needs to answer the main arguments (6 major concerns) in short and think about if the answer of each concern covered the explanation of most of the evidence. You don't need to write answer for each evidence because you can directly edit related sections in your paper. (I remember ICLR can let authors to edit paper last year in the rebuttal phase)
- Specify the exact protocol for Table 3 (how the subject is specified, which head produces L2 and AUC), report AP_in/out on VAT and ChildPlay, and report results using the GT head box as the only subject specification.
- Add a plain SFT baseline (Qwen3-VL-2B on HoliGaze-Corpus without queries), an option-only MCQ baseline, and latency/FLOPs against Gaze-LLE.
- If there is no space in rebuttal windows, all minor issues can be skipped.