- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors test if VLMs can follow human eye movements and glance around an image to gather more information, and evaluate if they do so similarly to humans. They show that training on human scan data can assist the model in choosing where to look and gathering information, and claim that this makes the model's representations able to better predict primate neural activity. They evaluate on Air-D, VQA-MHUG, Coco Search18 and Natural Design.
Critical strengths
- They present new findings connecting VLM vision to human vision, and try to create a setup in which they can be compared despite the vast methodological differences in human/biological versus transformer-based attention.
Critical weaknesses
- Novelty- while I do not believe their experiments are fully duplicated elsewhere, their claims are not aligned well with their experiments. It is not clear that their definition of active vision is meaningful (see questions). Other research, for example, has shown that VLMs might have internal "active" vision (Gaze Heads: How VLMs Look at What They Describe, Gandikota 2026). What is the motivation for this specific setup and definition of active learning, and why are the results meaningful?
- No clear focus and instead presents their work as a "bag of results"
- Is the goal to find if VLMs can teach us more about humans (L54-56) or to test if VLMs can perform active vision (L27)? For example, the experiments do not prove that VLMs do not do some sort of active vision internally and this is simply a poor interface for them to access.
- Disorganized experiments/prose - the graphs are difficult to read (Figure 2, table 1 fiure 3, figure 6) in relation to the claims that they are making. exp-1, exp2, etc should be properly labeled.
- Brain activity results are based on two monkeys only, and the correlations are quite small. I have no idea how significant this actually is and they do not compare it to to other types of finetuning.
- Overall, the ordering of the results and their organization must be fixed. The order and experiments do not clearly connect, and it is not obvious why they chose the specific experiments they did.
Questions for authors
- Is it possible that LLMs have different visual acuity with respect to gaussian blurring? (maybe because I believe they are trained on low resolution images at first) Did you try different levels of blur to see if they better approximate human performance? How did you justify your specific settings?
- "VLMs successfuly predict human fixation locations" (L200-240)- I am unsure how this evidence shows that, including table A7. Centre bias and chance seem to be naive baselines, and yet LLM performance is near them- how good are these? How good is it if we just do NN over the human dataset? Can you provide better intuition to what a strong method would score and justify it?
- Please clarify how the Human-between probability maps (Tables 2/A7) are made.
- How do you know the model does not simply have difficulty mapping the numerical grid to the image? For example, did you test if it can localize objects to the correct grid spot in general? As a human, this might be difficult to pinpoint exact locations, even though we can glimpse around a scene at ease.
- I do not see a clear related work section, and your intro does not carefully treat what work came before it and how it relates to what they do. While they have citations in introduction (L35-43, L53-54) these are not treated carefully and do they do not connect their work meaningfully to previous attempts, such as "GazeXplain" (Chen 2024) or V* (Wu 2023)
8 comments
Response to Reviewer 3 (Part 2/6: Continued Weaknesses 1 and 2)
- Relation to internal attention in transformer models.
We appreciate the reference to Gaze Heads. Gandikota and Bau show that particular attention heads track the image regions being described and that intervening on those heads can redirect the model’s descriptions (Gandikota & Bau, 2026). This comparison relates to the distinction in cognitive science between overt attention, which involves directing gaze toward a location, and covert attention, which involves selectively attending without shifting gaze (Munoz & Coe, 2011; Bisley et al., 2011). Internal attention over image tokens provides a conceptual analogue of covert attention: it can emphasize peripheral regions within an available observation without requesting a new fixation. However, we don’t consider transformer attention weights to be equal to biological attention. Our focus is the functional analogue of overt visual sampling and its coordination with task performance. Explicit fixations change the available visual evidence, while internal attention operates on the representations already available. Accordingly, our experiments characterize the models’ ability to coordinate overt sampling under the tested interface and constraints. We will revise broad statements about a lack of “active vision” to make this scope explicit.
To examine the relationship between internal attention and explicit fixation selection, we have now performed two new analyses, described in Appendix M, ‘Internal attention and explicit spatial choices.’”” First, we compared the attention map of each head in each layer with the model's output fixation probability map at the same step. We analyzed LOOK steps (the next requested fixation) and ANSWER steps (the reported target location in visual search) separately. Most heads were significantly aligned with the next requested fixation in both models. Correlations were strongest in the middle decoder layers, with the same heads showing alignment across datasets (Fig. A19). The alignment with the reported target location (ANSWER) was weaker but still widespread (Fig. A20). Fine-tuning on human behavior increased the number of aligned heads on every dataset and more than doubled the alignment in layer 15 in visual search (Fig. A21).
Second, we used the attention-centroid analysis of Haputhanthri et al. (2026) to compare the internal head attention map centroids with the incoming and outgoing fixations. The best heads of the base model tracked the current fixation more closely than the requested one. After fine-tuning, the best heads were closer to the requested location (Figs. A22–A24). We describe heads exhibiting this correspondence as "fixation-aligned heads." This designation reflects spatial association; the analysis does not establish that these heads causally control fixation selection.
Response to Reviewer 3 (Part 3/6: Continued Weaknesses 1 and 2)
- Paper organization
“Disorganized experiments/prose—the graphs are difficult to read … in relation to the claims that they are making. exp-1, exp2, etc. should be properly labeled.”
“Overall, the ordering of the results and their organization must be fixed. The order and experiments do not clearly connect, and it is not obvious why they chose the specific experiments they did.”
Briefly, our reasoning for the organization of the paper in the current version follows five stages that build on each other. First, we benchmark seven VLMs on two visual search and two VQA datasets and compare their task performance, stopping behavior, and fixation behavior with those of humans. Second, the differences found in the first stage motivate three controlled experiments to track the source of the errors models make. In each one, we hold the observation history fixed using human scanpaths and test a single decision, namely when to stop (WHEN), what to answer (WHAT), or where to look next (WHERE). These experiments show that the models can predict human fixations and answer accurately from human observations, but they still stop at different times than humans. Third, we ask whether the differences could be alleviated by aligning the models’ behavior with humans through fine-tuning the models on each of these components separately and reassessing all three aspects of behavior to test how supervising each component affects the full perception-action loop. Fourth, given the observed improvements from targeted fine-tuning experiments in stage 3, we fine-tune on full human behavior and test generalization to held-out trials and transfer between visual search and VQA. This yields a model whose behavior more closely matches that of humans, and that can serve as a starting point for studying active visual cognition. Finally, the subsequent neural analyses ask what changes internally as the model acquires this behavior. The encoding analysis examines whether behavioral fine-tuning improves prediction of recorded neural responses. The analysis of internal representations then examines evidence-sensitive dynamics across fixations: in later layers, we observe signatures resembling evidence accumulation after fine-tuning. This serves both as an investigation of what fine-tuning changed and as an example of how such models can be used to study relationships between internal computations and active visual behavior.
We understand that this structure has made it difficult to read through our manuscript. We are open to further improving the organization and would appreciate any suggestions from the reviewer.
Response to Reviewer 3 (Part 1/6: Weaknesses 1 and 2)
We thank the reviewer for their careful evaluation of the paper and constructive comments. We are encouraged that the reviewer recognizes our new findings connecting VLM vision to human vision. Below, we address each of the questions and weaknesses the reviewer raised. We understand that the review process takes significant time and energy from you, and for that reason, we kept all of our responses as brief as possible. For more detailed responses related to each comment, we refer the reviewer to dedicated new sections in the appendix. These additions will be incorporated into the relevant sections of the paper in the revision. We believe our responses address the reviewer's concerns and questions but remain open to further discussions.
We thank the reviewer for raising these connected questions. Broadly, our study is motivated by the need for model systems for studying active visual cognition in-silico. As outlined in the introduction, vision models have been transformative in studying the neural computations in the visual cortex. However, we currently lack similar models to study active vision where animals sample their environment according to their goals. Such models would provide a platform for mechanistic analysis and interventions, helping researchers in cognitive science and computational and systems neuroscience develop and test hypotheses about active visual cognition. Establishing whether they can perform the behavior of interest, and how closely that behavior resembles human behavior, is a necessary step toward using them to investigate its underlying computations.
Our setup follows an established approach to modeling biological active vision: visual information is acquired through successive fixations, with high resolution near fixation and reduced resolution in the periphery. Previous work has used such constraints to investigate human active vision (Gupta et al., NeurIPS 2021; Schwinn et al., TMLR 2022; Choi et al., NeurIPS 2023; Blauch et al., ICLR 2026). Each work usually estimates the retinal transformation of foveal-perioheral view differently. Ours closely resembles the method used in (Gupta et al., NeurIPS 2021).
The recent CanViT work likewise motivates active-vision foundation models that build representations through sequential, localized glimpses and integrate information across observations (Berreby et al., 2026). Its focus is on general-purpose visual representations and integration across viewing policies; our focus is on whether pretrained VLMs can coordinate sampling and stopping decisions in ways that resemble human behavior. Thus, the biological motivation for our definition and foveated setup is well established.
This motivation builds on the growing use of neural networks as computational models of cognition. For example, [Binz et al., 2025, Nature] demonstrates that fine-tuning a language model on human behavior across diverse cognitive tasks improves its behavioral predictions and the alignment of its internal representations with human neural activity. Related work uses VLMs to investigate visual cognition, including the binding problem and selective attention (Campbell et al., 2024; Wang et al., 2025). These developments motivate asking whether such models can also capture cognition when perception depends on an agent’s own actions.
As a first step in that direction, our study therefore asks whether pretrained VLMs possess the necessary components to coordinate the perception-action loop in active vision, and whether human behavioral supervision helps them do so. We recognize that this setting differs from their usual full-image processing setting. Nevertheless, our experiments show that they already possess substantial component capabilities: given human observation histories, they can predict relevant next-fixation locations and answer more accurately when provided with the observations available at the human answering step. Their difficulty lies partly in coordinating these capabilities, particularly deciding whether to gather more information or answer. Targeted behavioral experiments and fine-tuning identify these strengths and weaknesses, while fine-tuning on full behavior improves performance and behavioral correspondence on unseen trials.
Improvements also transfer between tasks, suggesting that the learned information-gathering behavior extends beyond the task used for fine-tuning. This results in a VLM that can actually perform active vision and can be used as a model system by researchers in NeuroAI, computational systems, and cognitive neuroscience and cognitive science to study the underlying neural computations of more ecological behaviours using it.