acceptodds
Under review as a conference paper at ICLR 2027

Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models

Abstract

Cross-modal associations are systematic pairings of features across modalities, such as the association of ‘bouba’ with round shapes and ‘kiki’ with sharp shapes. Prior work has compared humans and vision–language models (VLMs) on such associations, but often using different stimuli or tasks between humans and models. Here, we ask whether VLMs align with humans not only in choices, but also in where they look when making those choices. We study both VLMs and humans (𝑁 = 53), presenting them with the same stimuli, a pseudo-word and two images, and record participants’ choices and eye movements, which we release. We find choice alignment in a subset of generative VLMs, but their saliency matches human gaze less closely than a center-bias baseline, a fixed Gaussian at the center of each image. Fine-tuning small VLMs on human choices brings their choice alignment to the level of a human majority-vote reference on unseen words and images, yet their attention still matches human gaze less closely than this baseline. Training model attention on human gaze raises attention–gaze correlation without improving choice alignment, and a single average gaze map per image position raises it by a similar amount. Matching human choices, or even human gaze patterns, is therefore not sufficient evidence of human-aligned cross-modal processing.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
cross-modal association, the bouba-kiki effect, human-centered AI, model bias, empirical study, sound symbolism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GrzmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares humans and vision-language models on sound–shape associations using the same stimuli. The authors collect choice and eye-tracking data from 53 participants and evaluate 12 pretrained VLMs. They also fine-tune three smaller models using human choices and gaze. Some larger models show moderate agreement with human choices, but their saliency maps align poorly with human gaze. Fine-tuning improves choice agreement on unseen words and images, while gaze supervision improves spatial alignment without a detectable improvement in choice agreement. A fixed average-gaze control produces similar spatial correlations.

Critical strengths
  1. The study compares humans and VLMs using the same stimuli and collects both behavioral and eye-tracking data. This provides a useful basis for examining human–model differences beyond choice accuracy.

  2. The fine-tuning experiments include meaningful controls. In particular, the average-gaze condition shows that high attention–gaze correlation can be achieved without learning trial-specific gaze patterns. This is an important result for interpreting attention alignment.

Critical weaknesses
  1. The fine-tuned models may only have learned a phoneme rule. Test words use the same 12 phonemes as Train words, and these phonemes determine the intended category. The models’ agreement with participants (72.7–72.9%) equals that of always choosing the intended category (72.82%), and they rarely follow participants who deviate (11–13%). This suggests rule learning rather than human-like association. Held-out phonemes would give a stronger test.

  2. The attention–gaze comparison may not support the main conclusion. Human gaze here also reflects reading aloud and the gaze cascade. For simple shapes on coarse grids, a center Gaussian is hard to beat, and averaged attention or panel-level Grad-CAM are weak saliency measures. Without an inter-observer ceiling or an occlusion-based check, low alignment may reflect the readout rather than the model. The title and abstract should be more cautious.

Questions for authors
  1. How sensitive are the spatial alignment results to the choice of attention layers, heads, or attribution methods?
  2. Would the effects of gaze supervision remain consistent across different training seeds and gaze-loss weights?
  3. Section 3 reports 7,141 constructed pseudo-word trials, but the Train, Val, and Test sets contain 3,683 trials in total. Could the authors clarify how the remaining trials were handled and whether their exclusion affects the evaluation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Grzm

Thank you for your constructive feedback and comments. Below, we refer to the points under "Critical Weaknesses" as W1, W2, etc., and the questions as Q1, Q2, etc.

(W1) You wrote that the models may have just learned a phoneme rule. We agree that the fine-tuned models may have learned a mapping from phonemes to shape categories. We want to clarify how this relates to our claims. First, a phoneme-level rule is part of the phenomenon itself. Sound symbolism in humans can be described as specific phonemes/sounds eliciting specific shapes. Our pseudo-words are constructed from these rules, so a model applying a phoneme rule is not necessarily unlike humans in its choices. Second, this ambiguity motivated our paper. For the fine-tuned models, intended category association is higher than the choice agreement, so choice alignment alone cannot tell us whether a model has learned a rule or something closer to human-like association. Hence, we ask whether the reasoning behind a choice can be examined, and visual attention, compared against human gaze, is a first step. We agree that a held-out phoneme test would give a stronger test, which we will address in future work.

(W2) On reading aloud and the gaze cascade: We would like to make two clarifications. The word stimuli appeared on a separate slide before the image slide, and gaze is collected separately. Thus, reading the label is not part of the analyzed gaze. Additionally, the within-image correlation is computed separately for each image, so it does not depend on how gaze is split between the two images. This design addresses the gaze cascade effect (Section 4.1).

On the center baseline and inter-observer ceiling: We now report human-human gaze agreement (inter-observer ceiling). Participants’ gaze on the same image correlates at .786 on the Gemma/Qwen grid and .722 on Molmo’s. The gaze-supervised models exceed the center baseline and approach the human-human agreement ceiling.

On saliency measures: We agree that low alignment between humans and models could reflect the saliency readout rather than the model capabilities. Experimenting with different heads and different attribution methods, including occlusion-based checks, would be a valuable extension, which we leave for future work.

(Q1) You asked about the effects of different attention layers, heads, and attribution methods. For layers, we repeat the spatial analysis with three alternative layer aggregations in addition to the later half used in the paper: the final layer, the later quarter, and all layers.

fine-tunedconditionlater ½ layersfinal layerlater ¼ layersall layers
Gemma4 E4BZero-shot.349.327.357.398
Choice-only.308.294.298.401
+Gaze.733.729.731.706
+AvgGaze.733.730.732.707
Qwen3-VL-8BZero-shot.032−.060.036.021
Choice-only.065−.027.021.053
+Gaze.729.683.721.700
+AvgGaze.731.680.725.702
Molmo2-8BZero-shot.010.065−.006−.012
Choice-only.062.071.026.027
+Gaze.673.642.670.635
+AvgGaze.679.653.675.637

Our conclusions hold under every aggregation. No zero-shot or choice-trained model reaches its center baseline, and gaze supervision raises correlation consistently regardless of layers.

(Q2) You asked about the effects of different seeds and gaze-loss weights. Below, we show the results for three seeds (values separated by /).

ModelConditionHuman agreement ↑ (%)ρ ↑
Gemma4 E4BChoice-only72.72 / 73.59 / 73.20.308 / .284 / .338
+Gaze72.82 / 72.04 / 72.91.733 / .733 / .733
Qwen3-VL-8BChoice-only72.91 / 73.20 / 73.79.065 / .056 / .072
+Gaze72.82 / 71.84 / 73.79.729 / .731 / .726
Molmo2-8BChoice-only71.94 / 72.72 / 72.72.062 / .040 / .047
+Gaze73.11 / 72.14 / 72.91.673 / .673 / .677

The effect of gaze supervision is consistent across all three seeds and models. We agree that varying the gaze-loss weights would be a valuable extension, which we leave for future work.

(Q3) You asked about the difference in the number of trials. We will clarify this in Section 3.3. We split words and image pairs separately into training and test sets. Of the 162 pseudo-words, 100 are on the training and 62 are on the test. Of the 182 image pairs, 112 are on the training, and 70 are on the test. Validation is held out from the training, giving three sets:

  • Train (1,932 trials) and Val (721 trials) - training-side words shown with training-side image pairs
  • Test (1,030 trials) - test-side words shown with test-side image pairs, so both are unseen during training
  • Excluded (3,458 trials) - trials that combine sides; either a training-side word with a test-side image pair (1,717) or a test-side word with a training-side image pair (1,741) (assignments were randomized per participant)

The exclusion does not appear to bias the evaluation. Participants choose the intended category on 72.82% of Test trials, compared with 73.98% on all pseudo-word trials.

I appreciate the authors’ responses. However, my overall assessment remains unchanged.

Thank you for reading our responses and for your time throughout the review process. We will incorporate the revisions, clarifications, and analyses from our responses into the final version.

Reviewer_1ZvJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision-language models (VLMs) align with humans in both behavioral choices and visual attention during sound–shape association tasks, using matched stimuli and eye-tracking data from 53 human participants. Experiments across 12 VLMs show that some larger models exhibit human-aligned choices, but their spatial attention aligns with human gaze less closely than a simple center-bias baseline. Fine-tuning on human choices substantially improves behavioral alignment without improving gaze alignment, while gaze supervision improves spatial alignment without corresponding gains in choice accuracy, suggesting a dissociation between behavioral and spatial alignment.

Critical strengths
  1. clear and easy to understand with figures
  2. The paper evaluates humans and VLMs on identical stimuli, leveraging eye-tracking data from 53 participants and 8,208 trials to compare behavioral and spatial alignment.
  3. The choice-only, gaze-supervised, and average-gaze control experiments provide valuable evidence that behavioral alignment and spatial alignment do not necessarily improve together.
Critical weaknesses
  1. The conclusions rely heavily on attention and Grad-CAM, without sufficient robustness checks across layers, attribution methods, or models. It remains unclear whether the observed discrepancies reflect genuine differences in visual processing or limitations of the saliency methods.
  2. Lack of Human Gaze Consistency Analysis. While human choice reliability is reported, human–human gaze agreement is missing, making it difficult to determine whether model–human spatial differences exceed natural inter-participant variability.
  3. Limited Interpretability and Generalizability. The paper suggests that VLMs rely on global visual cues while humans attend to localized shape-informative regions, but provides limited qualitative or feature-level evidence supporting this interpretation. Moreover, it remains unclear whether the observed choice–attention dissociation generalizes beyond written pseudo-word and abstract-shape matching to broader multimodal reasoning tasks.
  4. Although motivated by sound–shape symbolism, the experiments present pseudo-words only as written text, without audio stimuli. This makes it difficult to disentangle genuine phonological associations from orthographic patterns or learned linguistic biases, limiting the interpretation of the findings as cross-modal sound symbolism.
  5. The results exhibit substantial prompt sensitivity: Switching to a neutral prompt changes the predicted choices for 33.23% and 20.55% of the 6,720 unique pseudo-word inputs for Gemma4-26B and Qwen3-VL-30B, respectively
Questions for authors
  1. Do the main findings remain consistent when using different attention layers or alternative attribution methods? How consistent are saliency maps across different VLMs?
  2. What is the human–human spatial gaze agreement, and how does it compare with the reported model–human alignment?
  3. Does the spatial alignment gap persist when considering only trials where VLMs and humans make the same choice, or does alignment differ substantially between matching and mismatching decisions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1ZvJ (1/2)

Thank you for the constructive feedback and comments. Below, we refer to the points under "Critical Weaknesses" as W1, W2, etc., the questions as Q1, Q2, etc.

(W1, Q1) You asked about the effects of different attention layers and attribution methods, and the consistency of saliency maps across VLMs. For layers, we repeat the spatial analysis with three alternative layer aggregations in addition to the later half used in the paper: the final layer, the last quarter, and all layers.

fine-tunedcondition½ layersfinal layerlast ¼ layersall layers
Gemma E4BZero-shot.349.327.357.398
Choice-only.308.294.298.401
+Gaze.733.729.731.706
+AvgGaze.733.730.732.707
Qwen8BZero-shot.032−.060.036.021
Choice-only.065−.027.021.053
+Gaze.729.683.721.700
+AvgGaze.731.680.725.702
Molmo8BZero-shot.010.065−.006−.012
Choice-only.062.071.026.027
+Gaze.673.642.670.635
+AvgGaze.679.653.675.637

Our conclusions hold under every aggregation. No zero-shot or choice-trained model reaches its center baseline, and gaze supervision raises correlation consistently regardless of layers.

We agree that experimenting with different attribution methods would be a valuable extension, which we leave for future work.

We also compare saliency maps between models on the same trials. Maps are area-resampled to a common grid, and we compute the mean within-image Spearman correlation, using the same attention readout as the main text.

ConditionGemma E4B↔Qwen 8BGemma E4B↔Molmo 8BQwen 8B↔Molmo 8B
Zero-shot.088 [.072,.106].064 [.042,.086].185 [.173,.196]
Choice-only.171 [.159,.184].147 [.127,.166].200 [.187,.212]
+Gaze.895 [.888,.900].875 [.871,.879].876 [.871,.880]
+AvgGaze.949 [.947,.951].909 [.907,.912].902 [.899,.905]

Zero-shot results agree at .064-.185, which is much lower than the human-human agreement. Adding Choice-only fine-tuning brings agreement up to .200. With gaze supervision, agreement goes up to .949, where the saliency maps are very similar between models.

(W2, Q2) You asked about the human-human spatial gaze agreement. We now report human-human gaze agreement on the same image. Agreement is .786 [.771, .802] in Gemma/Qwen’s grid, and is .722 [.707, .736] in Molmo’s grid.

Gemma4-E4BQwen3-VL-8BMolmo2-8B
Human-human (same image), new.786.786.722
Center-bias baseline.637.637.603
Zero-shot.349.032.010
Choice-only.308.065.062
+Gaze.733.729.673
+AvgGaze.733.731.679

Three conclusions follow. First, gaze is consistent across participants. Human-human agreement exceeds the center bias baseline, so participants share image-specific gaze patterns beyond a center bias. Second, pretrained (zero-shot) and choice-only trained models fall far below this human-human level. Third, +Gaze performs about the same as +AvgGaze, but remains below the same-image human-human agreement. As humans show consistent gaze behavior, gaze is a relevant signal, and low model-human alignment cannot be attributed to noisy gaze.

(W3) You commented on limited interpretability and generalizability. We agree that the global vs. local account is an interpretation, based on prior work showing vision transformers integrate information globally from early layers (Raghu et al., 2021), and we will present it as a hypothesis in Section 6. What our results establish is narrower: the models’ attention does not concentrate on the within-image regions that humans fixate on.

For generalizability, we agree that our evidence comes from a single task, and we will scope our conclusions accordingly. Findings in broader tasks are consistent with ours. In VQA, models and humans attend to different regions (Das et al., 2017), gains from human-attention supervision are reproduced by using random maps (Shrestha et al., 2020), and VLM visual attention can be disconnected from answer correctness (Liu et al., 2026). We will add this discussion to Section 6 and state that direct confirmation on broader tasks remains future work.

AuthorsOpenReview ↗

Response to Reviewer 1ZvJ (2/2)

(W4) You wrote that our experiment only uses text and does not have audio stimuli. We agree that our design does not separate phonological from orthographic contributions, and Section 3.1 frames the correspondence as one between written word form and shape for this reason. We will make this framing clearer and remove any wording that implies a purely auditory association. We note two points.

First, written presentation is part of how the effect is commonly studied in humans. Although the bouba-kiki effect in humans is usually attributed to audio-to-shape associations, prior experiments frequently provide human participants with both an acoustic and written form for pseudo-words, including Nielsen and Rendall (2013), on which our pseudo-words are based. Many prior works therefore cannot cleanly separate the contributions of orthography and sound in the bouba-kiki effect in humans. Second, an audio-only condition would not isolate phonology in literate human participants. A study by Cuskley et al. (2017), which explored the role of orthography, showed that auditory presentation does not fully remove the orthographic influence in literate participants, as letter curvature influences word-shape associations even when written word forms are never shown (these two points are supported by Kouwenhoven et al. (2025) as well). Since written form is already part of how sound-shape associations are tested in humans, we used written labels in our human study. Participants saw only the written label, with no audio played, and still reproduced the standard effect (canonical pseudo-words reach 84.71% intended-category agreement). Extending the same setting to audio-capable models, paired with an auditory human condition, is a next step, and we will add it to the discussion of future work.

(W5) You wrote that the results have prompt sensitivity. We agree that individual decisions depend on the prompt, and we report this in Section 4.2 and Appendix G. Our claims, however, rest on aggregate alignment, which remains positive across the two prompts. Under the neutral prompt, correlation with human round-choice rates on pseudo-words remains positive, and agreement with the participant’s actual choice on the same trial does not decrease. Qwen3-VL-30B’s majority match does fall from 0.68 to 0.57, which we report in Table 16. We believe the flip rate partly reflects how ambiguous pseudo-word-image pairs are. Humans also disagree on these items. Choosing the majority response of other participants predicts an individual’s choice in only 72.86% of Test trials, and about a quarter of human responses do not match the intended category. On such inputs, a model’s two options may be close in probability, so small changes in the prompt can change the choice, without changing the overall pattern. We will state explicitly that the individual decisions are prompt-sensitive while aggregate alignment remains positive. Future experiments can also evaluate what part of the prompt affects the individual trial changes.

(Q3) You asked whether the spatial agreement gap persists when only looking at trials where VLMs and humans make the same choice. We split the test trials by whether the model’s choice matches the participant’s choice on the same trial, and compare within-image attention-gaze correlation.

In Test trials, match (%) means the ratio of trials in which the human and machine made the same choice out of all test trials;

Model·ConditionMatch (%)Match ρMismatch ρΔρ [95% CI]
Gemma26B · Zero-shot58.45.282.278+.004 [−.030,.038]
Qwen30B · Zero-shot55.83.025.009+.016 [−.030,.061]
Gemma4B · Zero-shot52.91.351.348+.003 [−.029,.035]
Choice-only72.72.313.295+.018 [−.021,.057]
+Gaze72.82.732.734−.002 [−.024,.019]
+AvgGaze73.01.731.738−.006 [−.027,.014]
Qwen8B · Zero-shot48.54.029.035−.007 [−.052,.039]
Choice-only72.91.072.048+.024 [−.021,.069]
+Gaze72.82.728.731−.003 [−.024,.018]
+AvgGaze71.75.730.734−.005 [−.025,.016]
Molmo8B · Zero-shot50.00.017.003+.013 [−.026,.053]
Choice-only71.94.069.042+.027 [−.016,.070]
+Gaze73.11.675.670+.005 [−.019,.029]
+AvgGaze72.43.680.678+.003 [−.021,.026]

The spatial alignment gap persists even for trials where models and humans make the same choice. Alignment does not differ between matching and mismatching trials for any model or condition (all Holm-adjusted p = 1.000). We will add this analysis to the paper.

Thank you for the additional analyses. My concern has been addressed except

  1. The layer-aggregation and cross-model comparisons address part of W1/Q1, but robustness to alternative attribution methods within the same models remains untested, limiting interpretation of the spatial discrepancy as a difference in underlying visual processing.

  2. For W4, I appreciate the acknowledged phonology–orthography confound and would consider this concern addressed if the claims are consistently restricted to written word–shape associations.

  3. For W5, positive aggregate choice alignment under the neutral prompt is reassuring, but spatial alignment has not been evaluated under that prompt, leaving the prompt robustness of the choice–attention dissociation unestablished.

Reviewer_yHgeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work investigated whether VLMs show human-like cross-modal associations by aligning computational evaluation as closely as possible with a human experiment. To do so, the paper continues an established line of work and clearly expands it. The results corroborate earlier findings that showed that out-of-the-box VLMs do not display human-like associations on pseudowords. Extending these findings, the paper reveals that attention patterns do not align with human gaze and that neither fine-tuning on human choices nor human gaze results in human-aligned cross-modal processing in VLMs.

Critical strengths
  1. The work adds a human experiment that includes valuable human gaze data, closes the gap between human and computational data, and allows for careful comparisons. Though Kouwenhoven et al. 2025 compared against human data from Cwiek et al. 2022, the current human experiment is a clear contribution to the field that can potentially yield more human-like VLMs.
  2. The work builds on prior studies and is well grounded in the literature.
  3. The work is extensive in the number of experiments and evaluations. However, this sometimes clouds the paper's main message and makes it difficult to follow. The authors took great care in conducting careful evaluations.
Critical weaknesses
  1. The results (section 4.2) are written in a very dense manner. This makes it very difficult to grasp what I should take from them. Perhaps it would help to write a summary/recap at the end of Section 4. Overall, the paper lists many results but, in my opinion, fails to put them into context or explain the grander rationale.
  2. Why do the authors expect to find a cross-modal association in the first place? Section 5 has the potential to reveal whether explicit human alignment can improve such associations. But the current paper does not explain the rationale (beyond linking the model choices and attention). The potential of human gaze data could, in my opinion, help us better understand what differences there are between computational and human mechanisms, such that these differences can be used to steer computational training/vision paradigms.
  3. While informative, Table 2 currently merely states that random Gaussian noise is a better fit to explain model choices than human gaze data. Perhaps focusing on HOW/WHERE human gaze data differs from Gaussian noise would help explain what is so special about/why human gaze data is informative of human predictions.
  4. The writing is very dense (see phrasing notes) to the point that it hurts readability and clarity of the paper. As the authors acknowledge, they used generative AI to polish the writing. I would, perhaps controversially, suggest rewriting it without generative AI to improve readability.
  5. Tables are ambiguous due to their density and dependence on many different aspects of interpretation/abbreviations/lengthy captions. The captions introduce new metrics/measurements, which should be in the main text.
  6. The discussion is relatively short and falls short in embedding it in the work that is presented in the introduction/related work. How do the observed results relate to Alper & Averbuch-Elor, Verhoef et al., Kouwenhoven et al., Loakman et al., etc.? The discussion would benefit from addressing weakness 2. Additional points for discussion: Are ‘naturalistic’ images generated by Gemini subject to these structure/texture biases in VLMs? And are they easier to disentangle? Lines 457-458 on the nature of “good performance” of Gemma4-26B and Qwen3-VL-30B: “it more plausibly reflects statistical regularities linking written form, phonology, and visual features in training data.” These are the interesting bits of the discussion. Could the authors elaborate on this reasoning and explicate how this differs/aligns with the origins of human cross-modal associations?
  7. The limitations section is very limited. It should, at a minimum, include how these limitations can alter/influence the interpretation of the results.
Questions for authors
  1. “prior studies often stop short of a direct, matched comparison: they evaluate humans and VLMs on different stimuli or different tasks, or analyze them at different levels.” --> This is not entirely true; Kouwenhoven et al. (2025) also conducted a matched comparison with Cwiek et al. (2022).
  2. Table 1 provides a lot of information, but in doing so it is also somewhat ambiguous/difficult to understand what a reader should take from this. Why report the overall results? It seems unnecessary since this metric combines the pseudowords with the adjectives. The adjectives only act as a reference and should not be combined with the pseudowords, if I am not mistaken?
  3. To what extent do the authors know that the observed effects should be attributed to the object shapes and not to their textures in the generated images? Since texture is a salient feature for VLMs, it may influence the predictions and act as a superficial cue for models.
  4. Do the authors have a (preliminary) hypothesis as to why choice alignment appears only for larger VLMs?
  5. How many trials did the participants do? Were they familiar with the effect? If I am not mistaken, it seems that each participant judged all 176 items. Did they show fatigue? Or understand the underlying experimental question at some point?
  6. What does “removing the sound-symbolic framing does not remove the association” (line 254) mean? If I am not mistaken, this has not been explained before.
  7. What is the authors' rationale or motivation for closing the “spatial gap”?
  8. Section 5.1 : +GAZE: Is there consistency between participants’ gaze behaviour? That is: do different participants look at the same places for the same images? If this is not the case, how do the authors expect that gaze information can be used as an additional signal for models? If humans do not show consistent gaze behaviour, perhaps all that this information can add to a model is noise?

Textual notes:

  1. What does "side-locked" mean? Do the authors mean that the models show a consistent bias toward one side of the image? If so, I suggest using naturalistic wording that is common in the literature.
  2. “eager” forward pass… (line 216)? Please use "greedy decoding" or state that the temperature is set to 0.0.
  3. Lines 226 – 228: "For embedding models, we follow Kouwenhoven et al. (2025) and compute Grad-CAM maps (Selvaraju et al., 2017), using the fixed caption “The label for this image is {word}” on the 512 × 512 composed image input, keeping each model’s native spatial cells without upsampling." This is incorrect: Kouwenhoven et al. 2025 did not fix their caption.
  4. Lines: 247-248: difficult to read.
6 comments
AuthorsOpenReview ↗

Response to Reviewer yHge (1/2)

Thank you for your constructive feedback. Below, we refer to the points under "Critical Weaknesses" as W1, W2, etc., the questions as Q1, Q2, etc., and the textual notes as T1, T2, etc.

You wrote that the paper's density sometimes hurts readability (W1,4,5; T1,2,4). We agree, and we will rewrite the paper for readability. In particular, we will replace "side-locked" with “consistent bias toward one side of the image" and "eager forward pass" with "forward pass".

(W1) As suggested, we will add a summary at the end of Section 4.2:

On adjectives, most models align with human choices; on pseudo-words, a few larger models do, even without sound-symbolic cues in the prompt. Yet even these models' saliency does not match human gaze: human-like choices do not imply human-like attention. Section 5 examines whether training can bring the two together.

(W5) We will move metric definitions from the captions into the main text.

(Q2) We will simplify the tables. Table 1 will drop the Overall columns, and report canonical labels separately.

(W2) You asked why we expected to find a cross-modal association in the first place. Human languages have various sound-symbolic associations. When VLMs are trained on human-written captions paired with images, these associations are inherently present in their training data. Hence, we believe that VLMs can acquire that association. The question is whether they acquire it the same way humans do, which is why we examine both choices and attention.

(W3) You wrote:

Table 2 currently merely states that random Gaussian noise is a better fit to explain model choices than human gaze data.

We would like to make two clarifications. First, the center-bias baseline is not random noise but a fixed Gaussian placed at the center of the image. It is the same for every trial and uses no information about the word, the image, or the participant. We introduce this baseline because the center bias (fixation on the center of a stimulus) has been observed in previous eye-tracking studies (Tseng et al., 2009). Second, the row labeled "Center" does not measure the correlation between the fixed center map and model saliency; it measures the correlation between the fixed center map and human gaze. The takeaway is that human gaze correlates with the center map better than with model saliency.

Tseng PH, Carmi R, Cameron IG, Munoz DP, Itti L. Quantifying center bias of observers in free viewing of dynamic natural scenes. J Vis. 2009 Jul 9.

(W6) We will expand the discussion with connections to prior work, rationale, texture bias, and how humans and models differ. For embedding models, we replicate the near-chance pseudo-word performance reported by Kouwenhoven et al. (2025) and Verhoef et al. (2024). The strong associations reported by Alper and Averbuch-Elor (2023) do not appear in our forced-choice setting, except SigLIP-so400m. For generative VLMs, we see some dependence on model size, consistent with Loakman et al. (2024). Prior work tested whether models make the same choices as humans using CLIP embedding projections, label probabilities, Grad-CAM-based choices, and prompting. We extend their work by showing that models that make the same choices as humans attend to different places.

(W6, Q3) You asked about texture bias in VLMs. We agree that this should be examined so that the observed effects can be attributed to the shapes. We compared the intended category agreement (ICA) on line-based vs. Gemini-generated images (zero-shot, 7,141 pseudoword trials).

Line ICAGemini ICAΔ Gemini−Line
Human.744.736−.008
CLIP RN50.523.518−.006
CLIP ViT-B/32.506.503−.003
SigLIP-base.549.494−.055
SigLIP-so400m.553.577+.024
SigLIP2-base.504.491−.013
SigLIP2-so400m.506.500−.006
Gemma4-E4B.516.507−.008
Gemma4-26B.613.630+.017
Qwen3-VL-8B.515.517+.001
Qwen3-VL-30B.577.555−.021
Molmo2-4B.507.507.000
Molmo2-8B.538.527−.012

The association also appears on line-based images, which have no texture difference, so we see no evidence that texture in the Gemini images drives it.

You also asked how cross-modal associations in humans and models may differ or align. For adjectives, models may have learned the association from image-text pairs containing those words. For pseudo-words, models cannot rely on having seen the label, so they must use learned rules relating orthography and phonology to shapes. In literate humans, letter shape also influences word–shape associations (Cuskley et al., 2017). If the model learned a similar association between phonemes to shape, then it would exhibit sound-symbolic correlations, although models cannot access sound or articulation. Humans show the association before literacy (Maurer et al., 2006), pointing to sources that precede language development. Models plausibly acquire a second-hand record of the human bias. This could explain how choices can align while attention differs.

AuthorsOpenReview ↗

Response to Reviewer yHge (2/2)

(W7): We will expand the limitations section on how each limitation affects the interpretation of results.

Attention and Grad-CAM are attribution methods, so they do not give causal explanations. Part of the spatial misalignment between humans and models may reflect the limitations of the attribution methods rather than how models process the images. Our conclusions concern associations between writing and shape, not phonology alone. As the participants were English-speaking members recruited at one institution, the human references may reflect English orthography and phonology. Human references and model alignment may differ in other languages.

(Q1) You noted that Kouwenhoven et al. (2025) also conducted a matched comparison. Thank you for the correction. We agree that the original sentence understated the results of Kouwenhoven et al. (2025), who compared CLIP with the human results of Ćwiek et al. (2022); we will revise that sentence to acknowledge their contribution. We will make more explicit what our study adds to prior work: identical written stimuli for humans and models, comparison at the level of individual trials between humans and models, and comparison of attention as well as choices.

(Q4) You asked about why we think choice alignment appears only for larger VLMs. We hypothesize that the sound-symbolic association between pseudowords and images is present as a weak signal in all generative VLMs we tested, and that only in the larger generative models is it strong enough to outweigh a positional bias for one side of the frame. We note two points. First, the small models’ show a positional bias. On Test, Qwen3-VL-8B answers B on 96% of trials (Appendix L). After fine-tuning, all small models reach 92-94% intended category association, suggesting the association could be learned with little data. Second, with intended category agreement, the small models match adjectives nearly as well as the larger ones while doing poorly for pseudo-words. Alignment for pseudo-words has to come from weaker associations between word form and meaning, which may require more capacity to store than lexical meanings of real words. Because different model sizes can have different architecture, we present this as a hypothesis and will state it as such in the paper.

(Q5) You asked about details on the user study. Each participant had 186 trials, including canonical pseudo-words and adjectives (Appendix E). We did not explicitly ask whether participants knew about the effect or understand the underlying question before or during the study. We also did not measure fatigue, but had two breaks to mitigate fatigue.

(Q6) You asked about

removing the sound-symbolic framing does not remove the association (line 254).

We acknowledge that this is not clear. Our original prompt for the generative VLMs instructs the model to judge using “visual/sound-symbolic fit” (Appendix C). This explicitly cues the association and could inflate agreement. To check this, we rerun with a neutral prompt that mirrors the instruction we provided our human participants (“Choose the image that better matches this word”). “Removing the sound-symbolic framing” refers to using the neutral prompt for the generative VLMs. Using this neutral prompt, correlation with human round-choice rates remains positive, showing that the correlation does not depend on the cue. We will revise the paragraph to make this clear.

(Q7) You asked about the rationale for closing the spatial gap. As the reviewer mentioned in W2, if we understand the differences between computational and human mechanisms, we could steer models using human data. While model attention is not equal to causality or human saliency, it is a first step in understanding whether human-like processing is possible in a cross-modal task.

(Q8) You asked about the consistency between participant’s gaze. We now report human-human gaze agreement on the same image. Agreement is .786 [.771, .802] in Gemma/Qwen’s grid, and is .722 [.707, .736] in Molmo’s grid.

Gemma4-E4BQwen3-VL-8BMolmo2-8B
Human-human.786.786.722
Center-bias baseline.637.637.603
Zero-shot.349.032.010
Choice-only.308.065.062
+Gaze.733.729.673
+AvgGaze.733.731.679

Gaze is consistent across participants, and human-human gaze agreement exceeds the center bias baseline, so participants share image-specific gaze patterns beyond a center bias. As humans show consistent gaze behavior, low alignment between model saliency and human attention cannot be attributed to noisy gaze.

(T3) Thank you for the correction regarding Kouwenhoven et al (2025). We follow Kouwenhoven et al. (2025) in the Grad-CAM procedure, but not in caption choice. They evaluated ten captions, whereas our Grad-CAM uses one of them. Embedding-model choices still average over the ten captions. We will correct the sentence accordingly.

I want to thank the authors for their elaborate responses. Most of my concerns are adressed and I encourage the authors to include them in a revised version of their paper.

However, while interesting, I find the authors' rationale (w2) still rather weak. It mixes up sound - shape symbolism with word - shape associations. Which, admittedly, is mentioned in the paper, but the current resoponse is not much different from work that already exist. Still, the added human gaze data is very useful to further unravel the question of human-like pseudoword-shape associations.

For now, I will maintain my score since a "clear acceptance" feels just a bit too much for me.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.