acceptodds
Under review as a conference paper at ICLR 2027

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

Abstract

Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation–interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.

Primary area:
interpretability and explainable AI
Keywords:
vision language models, visual communication, emergent communication, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_piroOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how vision-language models (VLMs) interpret and communicate visual meaning. The authors show that simple visual symbols can influence model behavior on unrelated tasks, such as choosing recursion over iteration. These effects are not always reflected in the models’ explicit interpretations or reasoning traces. The paper further investigates visual communication between models through fixed symbol sets and free-form drawings, and compares model interpretations with human judgments. The results are interesting and raise useful questions about how VLMs acquire and use visual associations.

Critical strengths

The strongest contribution is the systematic discovery that simple, semantically lightweight visual symbols can induce large and model-specific behavioral changes on unrelated tasks.

The second major strength is the connection between visual steering and visual communication.

Critical weaknesses

The core phenomenon is convincing, but the interpretation in terms of visual meaning/communication occasionally goes beyond what the experiments directly establish.

The study does not identify the underlying representation or mechanism, and the authors themselves characterize the mechanism as an open question.

I would tone down the claims throughout. The evidence most clearly shows that VLMs respond to visual symbols in systematic ways that vary by model, and that their verbal reports do not accurately reflect these patterns.

Questions for authors
  1. Were the strongest steering symbols identified on one set of tasks and then evaluated on completely held-out tasks/symbols without re-selection? If so, how large are the effects on the held-out set?
  2. Can the authors provide confidence intervals or another uncertainty analysis for the reported steering effects, particularly the extreme values such as Δ = +1.0 obtained from n = 20 samples?
  3. How should we interpret the strongest effects among the 288-symbol library given the large number of symbol-condition comparisons? Were any corrections or preregistered selection procedures used?
  4. What happens when the sender and receiver are independently selected different model families? In particular, does a Claude sender successfully communicate with Gemini/GPT receivers using the same protocol?
  5. Given N = 7 participants, what are the participant-level distributions and statistical uncertainty around the reported ∼36% and ∼67% human accuracies?
  6. Do the authors have evidence distinguishing semantic associations from other image-encoder or tokenization-level effects? For example, can controlled visual transformations preserve the semantic identity of a symbol while systematically changing its steering behavior?
1 comment
AuthorsOpenReview ↗

Response to Reviewer piro

We thank the reviewer for their thoughtful review and questions, and are glad they find the core phenomena convincing and our investigation of visual communication valuable. Please find point-by-point responses below.

Weaknesses

  1. "...more conservative wording": We agree that our results contribute both structured associations/regularities and highlight nuances (model-specificity, context-sensitivity). We will revise the introduction, results, and discussion to make this intended claim more clear and to use more conservative wording.

Questions

  1. "...evaluated on completely held-out tasks": To evaluate transfer without re-selection, we fixed the top three steering symbols on inorder binary tree traversal for each model and analyzed their steering effect on five additional recursion vs. iteration problems (App. B.3.1). We observe transfer of selected symbols to several additional tasks (GPT & Claude) while also identifying substantial task dependence, possibly reflecting different coding strategy priors for different problems:
    • GPT: red heart, brown heart, blue heart increase the rate of recursion by +0.48 on average on preorder traversal, +0.63 on graph DFS, and +0.57 for postorder traversal (none of the 288 symbols steer at all on Fibonacci or binary search; most symbols have no effect on any problem).
    • Claude: pink heart, brown heart, purple triangle increase recursion by +1.0 on average on preorder traversal; postorder traversal and graph DFS are 100% baseline recursive; on Fibonacci and binary search, only purple triangle steers (+0.5; almost no symbols steer on these problems).
    • Gemini: cyan spiral, pink plus, and yellow checkmark show little consistent transfer across the additional problems.
  2. "...confidence intervals": Yes! We computed 95% confidence intervals for the steering effect (i.e. difference in recursion proportion from baseline), accounting for sampling uncertainty in both symbol and no-image conditions. For Claude and GPT’s top symbols (pink heart, brown heart, purple triangle; red heart, brown heart, blue heart), the +1.0 steering effect (20/20 recursive generations vs. 0/20 baseline) has a pointwise 95% interval of [+0.82, +1.0]. For Gemini’s top symbol (yellow checkmark), the +0.5 steering effect (18/20 recursive vs. 8/20 baseline) has an interval of [+0.13, +0.70]. We will add these to the revised paper.
  3. "How should we interpret the strongest effects": We fixed the 288-symbol library before running any evaluations, so top-symbol effects are estimates within this library. To address your point about multiple comparisons, we’ve compared each of the 288 symbols with the no-image baseline using two-sided Fisher exact tests and applied Holm correction within each model. At a significance level of 0.05, many symbols are statistically significant recursion steerers for Claude (82 symbols) and GPT (21 symbols). The analysis was inconclusive for Gemini’s symbols (none of them survived correction). We will add this analysis to the Appendix and reference it in the main text.
  4. "...different model families": This is a great suggestion. We didn’t evaluate communication across model families, as our model-to-model communication experiments were primarily aimed at assessing whether models “knew” their own associations. It would be very interesting to look at cross-model communication in future work, and we’ll mention this direction in the discussion.
  5. "...participant-level distributions": These two quantities (∼36% and ∼67%) come from the N=3 participants assigned to the symbol (vs. free form drawing) condition. The 95% participant-bootstrap intervals are [21.4%, 57.1%] and [33.3%, 100%] respectively. We report individual results in the table below. The study is meant to be exploratory, and the small sample size limits generalization. We will add this analysis to the Appendix. | Participant | Top steering symbols (acc.) | Sender selected symbols (acc.) | | ----- | ----- | ----- | | 1 | 21.4% | 66.7% | | 2 | 28.6% | 33.3% | | 3 | 57.1% | 100% |
  6. "...distinguishing semantic associations": We investigate this in Section 3.2 and Appendix B.2.3 through various image perturbations (pixel shuffling, rotation, resizing, changes in spatial location within the image). Pixel shuffling preserves the color while disrupting the shape and substantially reduces steering for Claude and Gemini. Rotation, size, and position can also change the steering effect without replacing the underlying shape. However, these findings do not conclusively distinguish semantic associations from image-encoder effects; for example, spatial position could itself affect a symbol’s interpreted meaning. Further distinguishing between associations and image-encoder artifacts would likely require identifying internal mechanisms of visual steering, which we leave for future work.
Reviewer_cwZrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual meaning and communication in VLMs. The authors show that simple visual symbols can influence model behavior on seemingly unrelated tasks, while the models' explicit interpretations and reasoning traces often differ from their observed behavior. The paper further explores model-to-model visual communication through symbol selection and free-form drawing, as well as its alignment with human interpretation.

Critical strengths
  1. The paper presents an interesting empirical finding that simple, non-adversarial visual symbols can substantially influence VLM behavior on unrelated tasks. The model-specific steering patterns are particularly intriguing.
  2. The experiments cover multiple frontier models and tasks, with useful control experiments involving text captions, random noise, and visual perturbations.
  3. The discrepancy between generated visualizations, explicit interpretations, and observed behavior provides an interesting direction for understanding multimodal models.
Critical weaknesses
  1. Questionable faithfulness evaluation. The paper uses aggregate steering effects to classify individual reasoning traces as faithful or unfaithful. However, population-level changes in output probabilities do not establish whether a visual cue influenced a particular generation, potentially undermining the reported unfaithfulness statistics.
  2. Limited robustness of visual steering. Appendix B.2.4 shows that steering effects are sensitive to prompt formulation and image placement. Moreover, the weak correlation between explicit and silent injection conditions raises questions about whether the observed associations reflect stable visual semantics or prompt-dependent behavior.
Questions for authors
  1. Faithfulness evaluation: How can aggregate steering effects justify faithfulness labels for individual reasoning traces? Could the authors provide stronger evidence of instance-level causal influence?
  2. Robustness: To what extent do visual steering associations remain consistent across different prompting conditions? How do the authors distinguish stable visual–semantic associations from prompt-dependent response biases?
3 comments
AuthorsOpenReview ↗

Response to Reviewer cwZr (Weakness 1 + Question 1)

We thank the reviewer for their thoughtful review and feedback! We’re glad that they found the results intriguing and highlighted the behavior–interpretation discrepancy as a promising direction for understanding multimodal models. As each question expands on a corresponding weakness, we address W1/Q1 and W2/Q2 jointly below.

Weakness / Question 1: Faithfulness Analysis

Thank you for prompting us to examine this analysis more closely. Our experiments measure how visual symbols change the distribution of solution strategies, whereas the original faithful/unfaithful labels implied causal judgments about individual generations. We agree with the criticism that those labels were too strong: the design does not provide individual counterfactual outcomes. Based on your and reviewer BPEM’s feedback, we reanalyzed the thinking traces:

  • Instead of labeling individual traces, we tried comparing two rates for each steering symbol: how often a trace output follows the symbol’s steering direction AND receives a disregard label, and how often the model picks that strategy with no image. The difference estimates a minimum % of generations where the symbol changes the strategy while the trace disregards it. Averaged over steering symbols, this is at least 8.3% for Gemini and 6.0% for GPT. To quantify uncertainty, we computed exact (Clopper–Pearson) confidence limits for each symbol's rate and for the baseline rates, with Bonferroni correction across all symbols. Under this conservative procedure the 95% lower bound is 0, so we consider this result only suggestive and inconclusive on its own.
  • We also computed the disregard rate among traces whose outputs match their symbol’s observed steering direction. These rates are 73.8% for Gemini and 21.2% for GPT (vs. 71.5% and 16.9% originally reported “unfaithful disregard” rates). Thus, the descriptive pattern persists, although these percentages describe co-occurrence and do not establish individual causal influence.

We will revise our claim to be narrower: models' verbal reports (stated interpretations and reasoning traces) do not reliably track the behavioral shifts that visual symbols induce. We also highlight that our results on models’ direct interpretations and behavioral shifts under silent injection provide complementary evidence for this claim, and that our broader findings of imperfect alignment between model verbal reports and behavior do not rely on individual trace labeling.

In the revision we will: (i) replace "faithful/unfaithful" with neutral descriptive categories (attribution / disregard / no mention) and note that the rates are descriptive rather than causal; (ii) add the bounded analysis above with its uncertainty; and (iii) revise the thinking trace analysis paragraph in Section 3.4 to state the narrower claim. In direct response to your question about instance-level evidence, these analyses provide population-level assessments; they do not identify which individual traces were causally influenced. We therefore report them as descriptive results and aggregate bounds, rather than evidence of individually verified unfaithfulness.

AuthorsOpenReview ↗

Response to Reviewer cwZr (Weakness 2 + Question 2)

Weakness 2 / Question 2: Robustness

Characterizing both the regularities and nuances of visual steering is one of the central aims of Section 3. Our findings support the following characterization: visual steering reflects structured, context-sensitive associations. Our prompt variation, image placement, and silent injection results demonstrate context sensitivity, and we explicitly note that symbols “appear to shift behavior to varying degrees rather than acting as fixed, language-like mappings” (line 177). Nevertheless, we also observe significant regularities in the effect that persist across contexts:

  • Prompt variations: Across the 24 shapes in the symbol library, steering profiles remain moderately-to-highly correlated with the default prompt across prompt paraphrases for all models (r = 0.6-0.94 under lexical paraphrasing, r = 0.54-0.68 under clause reordering). The third variation, which removes explicit choice language, largely suppresses steering for Claude and GPT, while Gemini retains a correlation of r = 0.77; silent injection profiles correlate weakly with explicit instruction profiles for all models (r <= 0.18). Thus, substantial consistency holds across these paraphrases when models are explicitly asked to choose a strategy. We additionally highlight that prompt bias alone would not explain the systematic difference we observed across different types of images.
  • Transfer across coding tasks: We fixed the top three recursion-steering symbols per model from inorder traversal and evaluated them on five additional problems (without re-selection). On preorder traversal, their mean effects were +1.00 for Claude and +0.48 for GPT. GPT’s symbols also transferred to graph DFS and postorder, while Claude’s purple triangle increased recursion on Fibonacci and binary search (+0.50). Gemini’s selected symbols showed little consistent transfer.
  • Multiple comparison correction: Comparing each symbol with its model’s no-image baseline using two-sided Fisher exact tests, 82 Claude symbols and 21 GPT symbols retain significant increases in recursion after Holm correction across all 288 comparisons within each model (family-wise α=0.05\alpha=0.05). No Gemini symbols survive correction.

We will revise the introduction, Section 3, and discussion to foreground both these regularities and their contextual boundaries, particularly Claude and GPT’s sensitivity to explicit strategy-choice instructions. We hope these findings motivate further work on the mechanisms underlying visual steering.

Thanks for your clear responses, my concerns are fully addressed. I will raise my rating from 2 to 3 while confidence remain 2.

Reviewer_BPEMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how simple visual symbols steer VLM behavior and how models generate images to communicate coding strategies, abstract concepts, and stories. Experiments across models, visual perturbations, communication games, and a small human study reveal discrepancies between behavioral steering, explicit interpretation, and human understanding.

Critical strengths

The systematic evaluation of 288 visual symbols across multiple models and tasks establishes an intriguing behavioral phenomenon beyond isolated examples. Comparing symbol interpretation, behavioral influence, and visual generation provides a useful framework for studying inconsistencies in multimodal communication.

Critical weaknesses

The faithfulness analysis equates negligible changes in strategy frequency with absence of visual influence and applies condition-level effects to individual traces, weakening its conclusions about unfaithful reasoning (Appendix B.8). Strong prompt and image-position dependence, four-option story recognition, and the seven-participant human study limit the broader claims about visual meaning and human–model divergence.

Questions for authors

How do the faithfulness results change when accounting for sampling uncertainty and baseline ceiling effects, rather than treating a fixed steering threshold as evidence of influence or its absence? Do the interpretation–behavior discrepancies persist when both evaluations use matched task contexts and image positions, given the sensitivity reported in Appendix B.2.4?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BPEM (weaknesses)

We thank the reviewer for their engaged review and constructive feedback. We’re glad to hear that the reviewer finds our work easy to follow and systematic, and the interpretation-behavior comparison a useful framework. Please find point-by-point responses to your concerns below.

Weaknesses

  1. ...faithfulness analysis: Thank you for prompting us to examine this analysis more closely. Our experiments measure how visual symbols change the distribution of solution strategies, whereas the original faithful/unfaithful labels implied causal judgments about individual generations. We agree with the criticism that those labels were too strong: the design does not provide individual counterfactual outcomes. Based on your and reviewer cwZr's feedback, we reanalyzed the thinking traces:

    • Instead of labeling individual traces, we tried comparing two rates for each steering symbol: how often a trace output follows the symbol’s steering direction AND receives a disregard label, and how often the model picks that strategy with no image. The difference tells us a minimum % of generations where the symbol changes the strategy while the trace disregards it. Averaged over steering symbols, this is at least 8.3% for Gemini and 6.0% for GPT. To quantify uncertainty, we computed exact (Clopper–Pearson) confidence limits for each symbol's rate and for the baseline rates, with Bonferroni correction across all symbols. Under this conservative procedure the 95% lower bound is 0, so we consider this result only suggestive and inconclusive on its own.
    • We also computed the disregard rate among traces whose outputs match their symbol’s observed steering direction. These rates are 73.8% for Gemini and 21.2% for GPT (vs. 71.5% and 16.9% originally reported “unfaithful disregard” rates). Thus, the descriptive pattern persists, although these percentages describe co-occurrence and do not establish individual causal influence.

    We will revise our claim to be narrower: models' verbal reports (stated interpretations and reasoning traces) do not reliably track the behavioral shifts that visual symbols induce. We also highlight that our results on models’ direct interpretations and behavioral shifts under silent injection provide complementary evidence for this claim, and that our broader findings of imperfect alignment between model verbal reports and behavior do not rely on individual trace labeling.

    In the revision we will: (i) replace "faithful/unfaithful" with neutral descriptive categories (attribution / disregard / no mention) and note that these rates are descriptive rather than causal; (ii) add the bounded analysis above with its uncertainty; and (iii) revise the thinking trace analysis paragraph in Section 3.4 to state the narrower claim. To respond specifically to the “absence of visual influence” point: we will relabel attribution without a detectable shift in strategy (e.g., a trace claiming a symbol steered for iteration when the baseline is 100% iterative) as "causally ambiguous attribution." Note that these cases still present a challenge for oversight, since a monitor may rely on the trace as a causal explanation of behavior.

  2. "...limit the broader claims": We agree that prompt and image-position sensitivity limit the generality of particular symbol–behavior associations. Characterizing this context sensitivity is itself part of our contribution. We also observe regularities across contexts; for example, steering profiles remain moderately-to-highly correlated across two of three prompt paraphrases for all models (r=0.6-0.94 under lexical paraphrasing, r=0.54-0.68 under clause reordering), and some shapes exhibit statistically significant, recurring steering tendencies across multiple coding problems. We will foreground both these regularities and their boundaries in the main text and qualify our broader claims accordingly. We hope these findings can open up more lines of inquiry on the precise mechanisms of visual steering.

    • We use four-option recognition for concepts and stories because recovering an exact intended target from an image alone is underdetermined (many interpretations may be compatible with the same visualization). This provides a controlled test of whether the image communicates information that distinguishes the intended target from alternatives.
    • Finally, we agree that the human study is exploratory, as noted in the main text; we will report confidence intervals and individual participant results in the Appendix (please also see our response to Reviewer piro’s Q5). We will audit our claims about human–model divergence to ensure they’re appropriately scoped.
AuthorsOpenReview ↗

Response to Reviewer BPEM (questions)

Questions

  1. "How do the faithfulness results change...": Please see our response to Weakness 1, where we report the uncertainty-adjusted analysis and explain how baseline ceilings motivate treating attribution without a detectable behavioral shift as causally ambiguous.
  2. "...matched task contexts": Thank you for this suggestion! The interpretation experiments we run in Section 3.4 and our main visual steering experiments (Section 3.1) both place the image at the beginning of the prompt. Following your suggestion to match the task context as well, we reran the interpretation experiments with the inorder binary tree traversal problem inserted after the image but before the direct interpretation instruction (“Look at the provided image. Does it suggest ‘recursion’ or ‘iteration’ as a coding strategy?”). We found that putting the coding problem in context increases the correlation between interpretations and behavior for Claude (r = 0.22 originally → 0.5), and only very slightly increases it for GPT (r = 0.31 → 0.36) and Gemini (r = 0.29 → 0.33). In all cases, there remains a substantial number of mismatches between interpretations and behavior. We will include this new result in the Appendix.

Thank you for the thoughtful reanalysis and clarifications, which address much of my concern about the original framing. I would encourage keeping the trace findings explicitly descriptive, given the zero lower confidence bound, and noting in the main text that matching task context substantially improves Claude’s interpretation–behavior agreement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.