- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares humans and vision-language models on sound–shape associations using the same stimuli. The authors collect choice and eye-tracking data from 53 participants and evaluate 12 pretrained VLMs. They also fine-tune three smaller models using human choices and gaze. Some larger models show moderate agreement with human choices, but their saliency maps align poorly with human gaze. Fine-tuning improves choice agreement on unseen words and images, while gaze supervision improves spatial alignment without a detectable improvement in choice agreement. A fixed average-gaze control produces similar spatial correlations.
Critical strengths
-
The study compares humans and VLMs using the same stimuli and collects both behavioral and eye-tracking data. This provides a useful basis for examining human–model differences beyond choice accuracy.
-
The fine-tuning experiments include meaningful controls. In particular, the average-gaze condition shows that high attention–gaze correlation can be achieved without learning trial-specific gaze patterns. This is an important result for interpreting attention alignment.
Critical weaknesses
-
The fine-tuned models may only have learned a phoneme rule. Test words use the same 12 phonemes as Train words, and these phonemes determine the intended category. The models’ agreement with participants (72.7–72.9%) equals that of always choosing the intended category (72.82%), and they rarely follow participants who deviate (11–13%). This suggests rule learning rather than human-like association. Held-out phonemes would give a stronger test.
-
The attention–gaze comparison may not support the main conclusion. Human gaze here also reflects reading aloud and the gaze cascade. For simple shapes on coarse grids, a center Gaussian is hard to beat, and averaged attention or panel-level Grad-CAM are weak saliency measures. Without an inter-observer ceiling or an occlusion-based check, low alignment may reflect the readout rather than the model. The title and abstract should be more cautious.
Questions for authors
- How sensitive are the spatial alignment results to the choice of attention layers, heads, or attribution methods?
- Would the effects of gaze supervision remain consistent across different training seeds and gaze-loss weights?
- Section 3 reports 7,141 constructed pseudo-word trials, but the Train, Val, and Test sets contain 3,683 trials in total. Could the authors clarify how the remaining trials were handled and whether their exclusion affects the evaluation?
3 comments
I appreciate the authors’ responses. However, my overall assessment remains unchanged.
Thank you for reading our responses and for your time throughout the review process. We will incorporate the revisions, clarifications, and analyses from our responses into the final version.
Response to Reviewer Grzm
Thank you for your constructive feedback and comments. Below, we refer to the points under "Critical Weaknesses" as W1, W2, etc., and the questions as Q1, Q2, etc.
(W1) You wrote that the models may have just learned a phoneme rule. We agree that the fine-tuned models may have learned a mapping from phonemes to shape categories. We want to clarify how this relates to our claims. First, a phoneme-level rule is part of the phenomenon itself. Sound symbolism in humans can be described as specific phonemes/sounds eliciting specific shapes. Our pseudo-words are constructed from these rules, so a model applying a phoneme rule is not necessarily unlike humans in its choices. Second, this ambiguity motivated our paper. For the fine-tuned models, intended category association is higher than the choice agreement, so choice alignment alone cannot tell us whether a model has learned a rule or something closer to human-like association. Hence, we ask whether the reasoning behind a choice can be examined, and visual attention, compared against human gaze, is a first step. We agree that a held-out phoneme test would give a stronger test, which we will address in future work.
(W2) On reading aloud and the gaze cascade: We would like to make two clarifications. The word stimuli appeared on a separate slide before the image slide, and gaze is collected separately. Thus, reading the label is not part of the analyzed gaze. Additionally, the within-image correlation is computed separately for each image, so it does not depend on how gaze is split between the two images. This design addresses the gaze cascade effect (Section 4.1).
On the center baseline and inter-observer ceiling: We now report human-human gaze agreement (inter-observer ceiling). Participants’ gaze on the same image correlates at .786 on the Gemma/Qwen grid and .722 on Molmo’s. The gaze-supervised models exceed the center baseline and approach the human-human agreement ceiling.
On saliency measures: We agree that low alignment between humans and models could reflect the saliency readout rather than the model capabilities. Experimenting with different heads and different attribution methods, including occlusion-based checks, would be a valuable extension, which we leave for future work.
(Q1) You asked about the effects of different attention layers, heads, and attribution methods. For layers, we repeat the spatial analysis with three alternative layer aggregations in addition to the later half used in the paper: the final layer, the later quarter, and all layers.
Our conclusions hold under every aggregation. No zero-shot or choice-trained model reaches its center baseline, and gaze supervision raises correlation consistently regardless of layers.
(Q2) You asked about the effects of different seeds and gaze-loss weights. Below, we show the results for three seeds (values separated by /).
The effect of gaze supervision is consistent across all three seeds and models. We agree that varying the gaze-loss weights would be a valuable extension, which we leave for future work.
(Q3) You asked about the difference in the number of trials. We will clarify this in Section 3.3. We split words and image pairs separately into training and test sets. Of the 162 pseudo-words, 100 are on the training and 62 are on the test. Of the 182 image pairs, 112 are on the training, and 70 are on the test. Validation is held out from the training, giving three sets:
The exclusion does not appear to bias the evaluation. Participants choose the intended category on 72.82% of Test trials, compared with 73.98% on all pseudo-word trials.