acceptodds
Under review as a conference paper at ICLR 2027

Closing the Perception-Action Loop with Human-Aligned Vision-Language Models

Abstract

Vision-language models (VLMs) have emerged as model systems for studying human visual cognition. However, their visual capabilities are primarily based on processing high-resolution images of full scenes, which is in sharp contrast to how humans process visual information. Human vision is inherently dynamic, relying heavily on eye movements to bring the high-resolution fovea onto different parts of a scene, so each fixation provides only a partial view. Here, we asked whether VLMs under similar partial observation constraints could efficiently choose where to look and use the information they acquired to solve visual question answering (VQA) and visual search (VS) tasks, and whether their behavior would resemble that of human participants. We found that, unlike humans, VLMs made poor use of successive fixations; their performance plateaued after the first few glimpses, and they frequently chose to answer prematurely. Fine-tuning on human fixation trajectories not only helped the models ground their answers in the available visual information across fixations, but also made their internal representations more predictive of neural responses in the primate brain while giving rise to cross-fixation evidence accumulation reminiscent of signals observed in the primate parietal cortex. Altogether, we show that pretrained VLMs largely lack the ability to engage in active vision, but that targeted fine-tuning on human behavior gives them a human-like perception–action loop and brings their representations into closer alignment with the brain. These properties make them strong candidates for studying the neural computations that underpin active vision in humans and non-human primates.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Vision-language models, human alignment, visual reasoning, active vision, visual search, visual question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WMvwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors test if VLMs can follow human eye movements and glance around an image to gather more information, and evaluate if they do so similarly to humans. They show that training on human scan data can assist the model in choosing where to look and gathering information, and claim that this makes the model's representations able to better predict primate neural activity. They evaluate on Air-D, VQA-MHUG, Coco Search18 and Natural Design.

Critical strengths
  • They present new findings connecting VLM vision to human vision, and try to create a setup in which they can be compared despite the vast methodological differences in human/biological versus transformer-based attention.
Critical weaknesses
  • Novelty- while I do not believe their experiments are fully duplicated elsewhere, their claims are not aligned well with their experiments. It is not clear that their definition of active vision is meaningful (see questions). Other research, for example, has shown that VLMs might have internal "active" vision (Gaze Heads: How VLMs Look at What They Describe, Gandikota 2026). What is the motivation for this specific setup and definition of active learning, and why are the results meaningful?
  • No clear focus and instead presents their work as a "bag of results"
    • Is the goal to find if VLMs can teach us more about humans (L54-56) or to test if VLMs can perform active vision (L27)? For example, the experiments do not prove that VLMs do not do some sort of active vision internally and this is simply a poor interface for them to access.
    • Disorganized experiments/prose - the graphs are difficult to read (Figure 2, table 1 fiure 3, figure 6) in relation to the claims that they are making. exp-1, exp2, etc should be properly labeled.
    • Brain activity results are based on two monkeys only, and the correlations are quite small. I have no idea how significant this actually is and they do not compare it to to other types of finetuning.
    • Overall, the ordering of the results and their organization must be fixed. The order and experiments do not clearly connect, and it is not obvious why they chose the specific experiments they did.
Questions for authors
  • Is it possible that LLMs have different visual acuity with respect to gaussian blurring? (maybe because I believe they are trained on low resolution images at first) Did you try different levels of blur to see if they better approximate human performance? How did you justify your specific settings?
  • "VLMs successfuly predict human fixation locations" (L200-240)- I am unsure how this evidence shows that, including table A7. Centre bias and chance seem to be naive baselines, and yet LLM performance is near them- how good are these? How good is it if we just do NN over the human dataset? Can you provide better intuition to what a strong method would score and justify it?
  • Please clarify how the Human-between probability maps (Tables 2/A7) are made.
  • How do you know the model does not simply have difficulty mapping the numerical grid to the image? For example, did you test if it can localize objects to the correct grid spot in general? As a human, this might be difficult to pinpoint exact locations, even though we can glimpse around a scene at ease.
  • I do not see a clear related work section, and your intro does not carefully treat what work came before it and how it relates to what they do. While they have citations in introduction (L35-43, L53-54) these are not treated carefully and do they do not connect their work meaningfully to previous attempts, such as "GazeXplain" (Chen 2024) or V* (Wu 2023)
8 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (Part 1/6: Weaknesses 1 and 2)

We thank the reviewer for their careful evaluation of the paper and constructive comments. We are encouraged that the reviewer recognizes our new findings connecting VLM vision to human vision. Below, we address each of the questions and weaknesses the reviewer raised. We understand that the review process takes significant time and energy from you, and for that reason, we kept all of our responses as brief as possible. For more detailed responses related to each comment, we refer the reviewer to dedicated new sections in the appendix. These additions will be incorporated into the relevant sections of the paper in the revision. We believe our responses address the reviewer's concerns and questions but remain open to further discussions.

  • Unclear motivation (Weaknesses 1 and 2):

“What is the motivation for this specific setup and definition of active learning, and why are the results meaningful?”

“Is the goal to find if VLMs can teach us more about humans … or to test if VLMs can perform active vision?”

We thank the reviewer for raising these connected questions. Broadly, our study is motivated by the need for model systems for studying active visual cognition in-silico. As outlined in the introduction, vision models have been transformative in studying the neural computations in the visual cortex. However, we currently lack similar models to study active vision where animals sample their environment according to their goals. Such models would provide a platform for mechanistic analysis and interventions, helping researchers in cognitive science and computational and systems neuroscience develop and test hypotheses about active visual cognition. Establishing whether they can perform the behavior of interest, and how closely that behavior resembles human behavior, is a necessary step toward using them to investigate its underlying computations.

Our setup follows an established approach to modeling biological active vision: visual information is acquired through successive fixations, with high resolution near fixation and reduced resolution in the periphery. Previous work has used such constraints to investigate human active vision (Gupta et al., NeurIPS 2021; Schwinn et al., TMLR 2022; Choi et al., NeurIPS 2023; Blauch et al., ICLR 2026). Each work usually estimates the retinal transformation of foveal-perioheral view differently. Ours closely resembles the method used in (Gupta et al., NeurIPS 2021).

The recent CanViT work likewise motivates active-vision foundation models that build representations through sequential, localized glimpses and integrate information across observations (Berreby et al., 2026). Its focus is on general-purpose visual representations and integration across viewing policies; our focus is on whether pretrained VLMs can coordinate sampling and stopping decisions in ways that resemble human behavior. Thus, the biological motivation for our definition and foveated setup is well established.

This motivation builds on the growing use of neural networks as computational models of cognition. For example, [Binz et al., 2025, Nature] demonstrates that fine-tuning a language model on human behavior across diverse cognitive tasks improves its behavioral predictions and the alignment of its internal representations with human neural activity. Related work uses VLMs to investigate visual cognition, including the binding problem and selective attention (Campbell et al., 2024; Wang et al., 2025). These developments motivate asking whether such models can also capture cognition when perception depends on an agent’s own actions.

As a first step in that direction, our study therefore asks whether pretrained VLMs possess the necessary components to coordinate the perception-action loop in active vision, and whether human behavioral supervision helps them do so. We recognize that this setting differs from their usual full-image processing setting. Nevertheless, our experiments show that they already possess substantial component capabilities: given human observation histories, they can predict relevant next-fixation locations and answer more accurately when provided with the observations available at the human answering step. Their difficulty lies partly in coordinating these capabilities, particularly deciding whether to gather more information or answer. Targeted behavioral experiments and fine-tuning identify these strengths and weaknesses, while fine-tuning on full behavior improves performance and behavioral correspondence on unseen trials.

Improvements also transfer between tasks, suggesting that the learned information-gathering behavior extends beyond the task used for fine-tuning. This results in a VLM that can actually perform active vision and can be used as a model system by researchers in NeuroAI, computational systems, and cognitive neuroscience and cognitive science to study the underlying neural computations of more ecological behaviours using it.

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 2/6: Continued Weaknesses 1 and 2)

  • Relation to internal attention in transformer models.

We appreciate the reference to Gaze Heads. Gandikota and Bau show that particular attention heads track the image regions being described and that intervening on those heads can redirect the model’s descriptions (Gandikota & Bau, 2026). This comparison relates to the distinction in cognitive science between overt attention, which involves directing gaze toward a location, and covert attention, which involves selectively attending without shifting gaze (Munoz & Coe, 2011; Bisley et al., 2011). Internal attention over image tokens provides a conceptual analogue of covert attention: it can emphasize peripheral regions within an available observation without requesting a new fixation. However, we don’t consider transformer attention weights to be equal to biological attention. Our focus is the functional analogue of overt visual sampling and its coordination with task performance. Explicit fixations change the available visual evidence, while internal attention operates on the representations already available. Accordingly, our experiments characterize the models’ ability to coordinate overt sampling under the tested interface and constraints. We will revise broad statements about a lack of “active vision” to make this scope explicit.

To examine the relationship between internal attention and explicit fixation selection, we have now performed two new analyses, described in Appendix M, ‘Internal attention and explicit spatial choices.’”” First, we compared the attention map of each head in each layer with the model's output fixation probability map at the same step. We analyzed LOOK steps (the next requested fixation) and ANSWER steps (the reported target location in visual search) separately. Most heads were significantly aligned with the next requested fixation in both models. Correlations were strongest in the middle decoder layers, with the same heads showing alignment across datasets (Fig. A19). The alignment with the reported target location (ANSWER) was weaker but still widespread (Fig. A20). Fine-tuning on human behavior increased the number of aligned heads on every dataset and more than doubled the alignment in layer 15 in visual search (Fig. A21).

Second, we used the attention-centroid analysis of Haputhanthri et al. (2026) to compare the internal head attention map centroids with the incoming and outgoing fixations. The best heads of the base model tracked the current fixation more closely than the requested one. After fine-tuning, the best heads were closer to the requested location (Figs. A22–A24). We describe heads exhibiting this correspondence as "fixation-aligned heads." This designation reflects spatial association; the analysis does not establish that these heads causally control fixation selection.

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 3/6: Continued Weaknesses 1 and 2)

  • Paper organization

“Disorganized experiments/prose—the graphs are difficult to read … in relation to the claims that they are making. exp-1, exp2, etc. should be properly labeled.”

“Overall, the ordering of the results and their organization must be fixed. The order and experiments do not clearly connect, and it is not obvious why they chose the specific experiments they did.”

Briefly, our reasoning for the organization of the paper in the current version follows five stages that build on each other. First, we benchmark seven VLMs on two visual search and two VQA datasets and compare their task performance, stopping behavior, and fixation behavior with those of humans. Second, the differences found in the first stage motivate three controlled experiments to track the source of the errors models make. In each one, we hold the observation history fixed using human scanpaths and test a single decision, namely when to stop (WHEN), what to answer (WHAT), or where to look next (WHERE). These experiments show that the models can predict human fixations and answer accurately from human observations, but they still stop at different times than humans. Third, we ask whether the differences could be alleviated by aligning the models’ behavior with humans through fine-tuning the models on each of these components separately and reassessing all three aspects of behavior to test how supervising each component affects the full perception-action loop. Fourth, given the observed improvements from targeted fine-tuning experiments in stage 3, we fine-tune on full human behavior and test generalization to held-out trials and transfer between visual search and VQA. This yields a model whose behavior more closely matches that of humans, and that can serve as a starting point for studying active visual cognition. Finally, the subsequent neural analyses ask what changes internally as the model acquires this behavior. The encoding analysis examines whether behavioral fine-tuning improves prediction of recorded neural responses. The analysis of internal representations then examines evidence-sensitive dynamics across fixations: in later layers, we observe signatures resembling evidence accumulation after fine-tuning. This serves both as an investigation of what fine-tuning changed and as an example of how such models can be used to study relationships between internal computations and active visual behavior.

We understand that this structure has made it difficult to read through our manuscript. We are open to further improving the organization and would appreciate any suggestions from the reviewer.

Reviewer_hR9ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study investigates whether Vision-Language Models (VLMs) can serve as effective models of human active vision, moving beyond traditional static image processing to dynamic, sequential observation. The authors benchmark various VLMs on VQA and VS tasks, finding that while base models tend to answer prematurely with scanpaths that diverge from human behavior, fine-tuning on human fixation trajectories significantly improves their sampling strategies. Furthermore, the fine-tuned models exhibit evidence accumulation dynamics and show better alignment with primate neural responses, suggesting that learning from human behavior enhances their biological plausibility and cognitive modeling capabilities.

Critical strengths

The paper is overall easy to follow.

Critical weaknesses
  1. Novelty and Motivation: The core premise, that VLMs underperform in human-like active vision settings but improve after fine-tuning, feels somewhat expected. Since standard VLMs are not explicitly designed or pre-trained for sequential, foveated processing, it is natural that they would initially struggle and subsequently benefit from task-specific fine-tuning. The authors should better articulate why using VLMs in this "unnatural" way provides unique insights beyond confirming this baseline expectation.

  2. Validity of Claims: Certain conclusions appear overreached given the presented evidence. For instance, the claim that behavioral fine-tuning increases neural predictivity is not strongly supported by the results in Figure 5. The connection between the observed metrics and actual neural alignment needs more rigorous justification or additional analysis to be convincing.

  3. Practical Impact: While the paper includes an extensive set of experiments, the broader implications for building better real-world models remain unclear. The authors should clarify how these findings translate into practical advancements or actionable guidelines for developing more robust and efficient vision systems in applied settings.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (Part 1/2: Weakness 1 & 3): Motivation and Implications of our study

We thank the reviewer for their careful evaluation of the paper and constructive comments. We are encouraged that the reviewer finds our paper clear and easy to follow. Below, we address each of the questions and weaknesses the reviewer raised. We understand that the review process takes significant time and energy from you, and for that reason, we kept all of our responses as brief as possible. For more detailed responses related to each comment, we refer the reviewer to dedicated new sections in the appendix. These additions will be incorporated into the relevant sections of the paper in the revision. We believe our responses address the reviewer's concerns and questions but remain open to further discussions.

  • Novelty and motivation (Weakness 1) & Broader implications (Weakness 3):

We thank the reviewer for raising this concern. Broadly, our study is motivated by the need for model systems for studying active visual cognition in-silico. As outlined in the introduction, vision models have been transformative in studying the neural computations in the visual cortex. However, we currently lack similar models to study active vision where animals sample their environment according to their goals. Such models would provide a platform for mechanistic analysis and interventions, helping researchers in cognitive science and computational and systems neuroscience develop and test hypotheses about active visual cognition.

We agree that sequential, foveated viewing differs from standard VLM training settings, and that difficulties followed by improvement through fine-tuning are plausible expectations. Our questions concern which capabilities already exist, what prevents their coordination into a perception–action loop, and whether behavioral fine-tuning can yield a model whose internal computations can be studied across visual tasks. Developing more robust or computationally efficient vision systems for real-world deployment is not the goal of this study. The improvements we report concern performance and behavioral and neural alignment within the active-vision settings we evaluate. Appendix N, “Related Works,” now places this motivation within the growing use of LLMs and VLMs as models of cognition, and positions our approach within existing computational models of active vision. In this section, we have made this scope explicit and distinguished the demonstrated improvements in active-vision behavior from engineering efforts to reduce visual computation through selective processing (e.g., AutoGaze, Shi et al., 2026; GazeVLM, Chen & Qi, 2025).

This motivation builds on the growing use of neural networks as computational models of cognition. For example, [Binz et al., 2025, Nature] demonstrates that fine-tuning a language model on human behavior across diverse cognitive tasks improves its behavioral predictions and the alignment of its internal representations with human neural activity. Related work uses VLMs to investigate visual cognition, including the binding problem and selective attention (Campbell et al., 2024; Wang et al., 2025). These developments motivate asking whether such models can also capture cognition when perception depends on an agent’s own actions.

This question is particularly relevant given the growing interest in ecological neuroscience, which emphasizes studying perception, cognition, and action together in relation to the opportunities for action available to an organism (Angelaki et al., 2026). An important step in this direction is to extend VLMs from models of how humans respond to fully presented images to models of how humans actively gather visual information. A key aspect of this problem is that the agent’s fixation choices determine which parts of the scene are seen at high resolution. As a result, although our scenes are static, the information available to the model is controlled by its own behavior. The foveated setting therefore reflects the biological constraints of the behavior we aim to model, even though it differs from the models’ usual training conditions.

Our longer-term goal is to move toward foundation models of active vision that researchers can use to investigate computations underlying visual cognition across tasks. Broadly pretrained VLMs offer a starting point because they bring extensive visual and semantic knowledge to different tasks within a shared model. Establishing and improving their behavioral correspondence with humans is a step toward using them for mechanistic analysis and interventions: examining how fixation selection, integration of successive observations, and stopping decisions interact. This is the scientific motivation for our behavioral experiments and fine-tuning, and for studying the internal computations that accompany the acquisition of active visual behavior.

AuthorsOpenReview ↗

Response to Reviewer 2 (Part 2/2: Weakness 2)

  • Validity of claims (Weakness 2):

We thank the reviewer for this comment. We agree that the visual comparison in Figure 5 should be accompanied by statistical tests and an assessment of the reliability of the neural measurements. We have therefore conducted two additional analyses, added in (Appendix O “Neural Predictivity: Statistical Tests, Noise Ceilings, and Targeted Fine-tuning”) and clarified the scope of our interpretation.

First, we compared the fine-tuned model with the base model using two-sided paired t-tests on the same neurons' predictivity scores, separately for each layer and brain region, with Holm correction for multiple comparisons. Fine-tuning significantly increased predictivity in the later layers of both OFC and VPA, whereas differences in the earlier layers were not significant (Fig. A25; exact p-values in its caption). These results support an improvement concentrated in later layers. Second, we estimated noise ceilings to contextualize the absolute correlations. Within each recording session, we split the trials into two halves, correlated each neuron's condition-averaged responses across the two halves, and applied the Spearman–Brown correction to obtain a correlation ceiling, using the same condition definitions as in the paper (Fig. A25). Both models reach a large fraction of the estimated ceiling in their best layers, so the absolute correlations should be interpreted relative to the reliability of the recordings. Fine-tuning raises predictivity mainly in the later layers (Fig. A25). These ceilings refer to correlations, not to percentages of neural variance explained. Our neural-predictivity measurement method follows the broad neural-encoding framework used in Centaur (Binz et al., 2025) and in Reason to Play, which predicts human fMRI responses from reasoning-model representations (Botos et al., 2026). The recording modalities and implementation details differ across these studies, so their absolute scores are not directly comparable.

We appreciate the reviewer’s concern about broader interpretations of “neural alignment.” Our conclusion is that behavioral fine-tuning improves the ability of later-layer representations to predict recorded macaque OFC and VPA responses under this encoding analysis, without neural supervision during behavioral fine-tuning (L21–24 and 26–29 in the Abstracts, L72–73 in the Introduction, L392–398 and L402–409 in the results, and L472–473 in the Discussion in the paper). This provides evidence of increased neural predictivity within the studied task and populations and we do not claim complete representational alignment or identical biological and artificial computations. We will revise the wording accordingly to clearly state this.

Reviewer_Fwg3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether vision language models (VLMs) could behave more like human in visual information understanding by enforcing the similar constraints: only a particular fixation location is clearly visualized simulating the human retinal view. Under this setting, VLMs are asked to start from the central fixation location, then, move to other locations until they are confident to respond. Experiments show that VLMs answer early leading to less accurate response although they can simulate the human fixation paths. Further analysis provides that stopping condition is different from human even if constrained to follow the same path and that the stopping decision is critical. Finally, this work shows that fine-tuning is necessary to close the gap.

Critical strengths
  • It is an interesting study in that VLMs are instructed to follow the human behavior sharing the similar human retinal view. They are asked to move the fixation location to decide when to stop and provide responses.
  • Experiments are well designed and the findings sound interesting enough for further studies to understand the behavior of VLMs.
Critical weaknesses
  • No headroom analysis: This work should provide the upper bound of the model performance under two conditions, one without any constraints and the other under the optimal fixation location after exploring all locations in a grid. Otherwise, the experiments mix the decision of the termination condition and the next move location, together with the vision understanding capacity.
  • Object location bias: This work does not differentiate the object location in an image, which might have an impact since the fixation location starts from the central position.
  • Comparisons with attention mechanism: Since VLMs often involve image tokens with attention mechanism to focus on particular tokens when generating response, it would be better to also investigate the attention weights to see how the explicit fixture will be different with the underlying mechanism.
Questions for authors
  • I'd expect further headroom analysis to see whether the current investigation, e.g., fixation location decision, reach the upper bound or not.
  • Additional location bias should be investigated by differentiating the experimental results by the object location, e.g., center/top/bottom/left/right coordinate, at least.
  • If possible, I'd like to understand the implication of this and future studies, why do we need to close the gap by simulating the human behavior, since vision model might be able to understand the whole information at once without simulating the retina view. If the task completion is the target, a model is not required to follow the same situation.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (Part 1/3: Weaknesses and Questions 1 and 2): Headroom Analysis and Object Location Bias

We thank the reviewer for their careful evaluation of the paper and constructive comments. We are encouraged that the reviewer finds our work an interesting study with well-designed experiments providing findings that sound interesting. Below, we address each of the questions and weaknesses the reviewer raised. We understand that the review process takes significant time and energy from you, and for that reason, we kept all of our responses as brief as possible. For more detailed responses related to each comment, we refer the reviewer to dedicated new sections in the appendix. We believe our responses address the reviewer's concerns and questions but remain open to further discussions.

  • Headroom analysis for determining the upper bound of performance (Weakness 1 & Question 1):

Thanks for suggesting this analysis, which is a good complement to our behavioral experiments, testing task performance without the effects of fixation selection and stopping behavior. We measured accuracy in three additional conditions on a subset of 20 held-out trials per dataset for each model, and report the results in Appendix K (Full-image and best-single-view accuracy). In the center-view condition, the model answers from the initial center view alone. In the full-image condition, it answers from the full, unfoveated image in one shot. In the best-single-view condition (the optimal fixation location), we test whether the model can answer correctly from any single retinal view across the grid. We provide one retinal view at a time in independent queries, starting at the center and testing successive grid locations until the answer is correct or all locations have been tried. The trial is scored as correct if at least one view leads to the correct answer. So, the best single view is the one that leads to the correct response when supplied to the model as the only observation, and if no such view exists, the trial is incorrect.

The best-single-view condition leads to near-perfect accuracy in many cases, often exceeding full-image performance (Table A14). This indicates that the models can perform the task when shown an informative observation. These findings complement our task-competence experiment (Experiment 2, Fig. 3b), where providing the human-matched sequence of observations and requiring an answer at the human answering step improved accuracy, although a gap to human performance remained. Together, these results establish an upper bound on the model performance and suggest that the models already possess substantial task and visual-understanding capabilities, reinforcing our finding that acquiring suitable observations and deciding when to answer contribute to the performance limitations in the free perception-action loop.

  • Object location bias (Weakness 2 and Question 2):

The reviewer raises an interesting question about whether the models’ performances are biased by object locations. In our original results, we followed the protocol used in the human psychophysics experiments where every participant starts from a center fixation. To address this point, we analysed whether the accuracy consistently changes as the object of interest, e.g., the target in visual search, is closer to the center.

We added Appendix L (Target Eccentricity) to address this question. Humans and models start every trial at the image center. We define target eccentricity as the distance between the target center and the image center, and perform three analyses on the two visual search datasets for humans and all models. First, both humans and models make more fixations before a correct answer when the target is farther from the center (Fig. A15). Second, for humans and most models, correct trials have targets closer to the center than incorrect trials (Fig. A16). Thus, the central start makes near targets easier to find for both humans and models. Third, model-human scanpath similarity and next-fixation scores decrease with target eccentricity, and those of the center-bias baseline decrease by a similar amount (Figs. A17–A18). Information Gain (IG) measures the gain over the center-bias baseline, and it does not depend on target eccentricity. Therefore, far targets lower the similarity of any center-anchored scanpath to humans, but the models' similarity to humans beyond the center bias does not depend on where the target is.

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 2/3: Weakness 3): Relation between gaze locations and internal attention maps

  • Relation between gaze locations and internal attention maps (Weakness 3):

We thank the reviewer for suggesting a comparison between explicit fixation selection and the model’s internal attention. This comparison relates to the distinction in cognitive science between overt attention, which involves directing gaze toward a location, and covert attention, which involves selectively attending without shifting gaze (Munoz & Coe, 2011; Bisley et al., 2011). Internal attention over image tokens provides a conceptual analogue of covert attention, which can emphasize peripheral regions within an available observation without requesting a new fixation. However, we don’t consider transformer attention weights to be equal to biological attention. Our study focuses on the functional analogue of overt attention, asking whether VLMs can coordinate a perception-action loop by choosing where to acquire new visual information. Explicit fixations change the available visual evidence by bringing different regions into high resolution, while internal attention operates on the representations already available.

To address this point, we performed two analyses, described in Appendix M (Internal attention and explicit spatial choices). First, we compared the attention map of each head in each layer with the model's output fixation probability map at the same step. We analyzed LOOK steps (the next requested fixation) and ANSWER steps (the reported target location in visual search) separately. Most heads were significantly aligned with the next requested fixation in both models. Correlations were strongest in the middle decoder layers, with the same heads showing alignment across datasets (Fig. A19). The alignment with the reported target location (ANSWER) was weaker but still widespread (Fig. A20). Fine-tuning on human behavior increased the number of aligned heads on every dataset and more than doubled the alignment in layer 15 in visual search (Fig. A21).

Second, we used the attention-centroid analysis of Haputhanthri et al. (2026) to compare the internal head attention map centroids with the incoming and outgoing fixations. The best heads of the base model tracked the current fixation more closely than the requested one. After fine-tuning, the best heads were closer to the requested location (Figs. A22–A24). We describe heads exhibiting this correspondence as "fixation-aligned heads." This designation reflects spatial association; the analysis does not establish that these heads causally control fixation selection.

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 3/3: Question 3): Implications of our study

  • Implications of our study (Question 3):

We thank the reviewer for raising this important question. We agree that the motivation behind simulating human behavior in these tasks was mentioned in passing and was not extensively discussed. We also agree that reproducing human sampling constraints is unnecessary if the sole objective is task completion with unrestricted visual input. Broadly, our study is motivated by the need for model systems for studying active visual cognition in-silico. As outlined in the introduction, vision models have been transformative in studying the neural computations in the visual cortex. However, we currently lack similar models to study active vision where animals sample their environment according to their goals. Such models would provide a platform for mechanistic analysis and interventions, helping researchers in cognitive science and computational and systems neuroscience develop and test hypotheses about active visual cognition. This motivation builds on the growing use of neural networks as computational models of cognition. For example, [Binz et al., 2025, Nature] demonstrates that fine-tuning a language model on human behavior across diverse cognitive tasks improves its behavioral predictions and the alignment of its internal representations with human neural activity. Related work uses VLMs to investigate visual cognition, including the binding problem and selective attention (Campbell et al., 2024; Wang et al., 2025). These developments motivate asking whether such models can also capture cognition when perception depends on an agent’s own actions.

This question is particularly relevant given the growing interest in ecological neuroscience, which emphasizes studying perception, cognition, and action together in relation to the opportunities for action available to an organism (Angelaki et al., 2026). An important step in this direction is to extend VLMs from models of how humans respond to fully presented images to models of how humans actively gather visual information. A key aspect of this problem is that the agent’s fixation choices determine which parts of the scene are seen at high resolution. As a result, although our scenes are static, the information available to the model is controlled by its own behavior.

As a first step in that direction, our study therefore asks whether pretrained VLMs possess the necessary components to coordinate the perception-action loop in active vision, and whether human behavioral supervision helps them do so. Improvements also transfer between tasks, suggesting that the learned information-gathering behavior extends beyond the task used for fine-tuning. This results in a VLM that can actually perform active vision and can be used as a model system by researchers in NeuroAI, computational systems, and cognitive neuroscience and cognitive science to study the underlying neural computations of more ecological behaviours using it.

Models that can accurately replicate behaviour and neural activity can in turn be used to generate hypotheses about neural computation. For example, in our work we show that as the model is aligned with humans, its internal dynamics develop gradual, evidence-sensitive increases in the preference to answer. Such response patterns had not been previously observed during visual search, but they resemble neural signatures of evidence accumulation during decision making reported in primate LIP (Hanks et al., 2014; Steinemann et al., 2024).

Appendix N, “Related Works,” now places this motivation within the growing use of LLMs and VLMs as models of cognition, and positions our approach within existing computational models of active vision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.