- Rating
- 3
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how vision-language models (VLMs) interpret and communicate visual meaning. The authors show that simple visual symbols can influence model behavior on unrelated tasks, such as choosing recursion over iteration. These effects are not always reflected in the models’ explicit interpretations or reasoning traces. The paper further investigates visual communication between models through fixed symbol sets and free-form drawings, and compares model interpretations with human judgments. The results are interesting and raise useful questions about how VLMs acquire and use visual associations.
Critical strengths
The strongest contribution is the systematic discovery that simple, semantically lightweight visual symbols can induce large and model-specific behavioral changes on unrelated tasks.
The second major strength is the connection between visual steering and visual communication.
Critical weaknesses
The core phenomenon is convincing, but the interpretation in terms of visual meaning/communication occasionally goes beyond what the experiments directly establish.
The study does not identify the underlying representation or mechanism, and the authors themselves characterize the mechanism as an open question.
I would tone down the claims throughout. The evidence most clearly shows that VLMs respond to visual symbols in systematic ways that vary by model, and that their verbal reports do not accurately reflect these patterns.
Questions for authors
- Were the strongest steering symbols identified on one set of tasks and then evaluated on completely held-out tasks/symbols without re-selection? If so, how large are the effects on the held-out set?
- Can the authors provide confidence intervals or another uncertainty analysis for the reported steering effects, particularly the extreme values such as Δ = +1.0 obtained from n = 20 samples?
- How should we interpret the strongest effects among the 288-symbol library given the large number of symbol-condition comparisons? Were any corrections or preregistered selection procedures used?
- What happens when the sender and receiver are independently selected different model families? In particular, does a Claude sender successfully communicate with Gemini/GPT receivers using the same protocol?
- Given N = 7 participants, what are the participant-level distributions and statistical uncertainty around the reported ∼36% and ∼67% human accuracies?
- Do the authors have evidence distinguishing semantic associations from other image-encoder or tokenization-level effects? For example, can controlled visual transformations preserve the semantic identity of a symbol while systematically changing its steering behavior?
Response to Reviewer piro
We thank the reviewer for their thoughtful review and questions, and are glad they find the core phenomena convincing and our investigation of visual communication valuable. Please find point-by-point responses below.
Weaknesses
Questions