acceptodds
Under review as a conference paper at ICLR 2027

They Know Where It Points, But Not What It Means: The Causal Grammar of Relational Visual Prompts

Abstract

Graphical cues can identify targets and express relations. We test whether vision-language models use an arrow to attribute speech after locating its endpoint. Across 450 paired conditions from 150 scenes, five open VLMs locate endpoints 8.5–13.1 percentage points more accurately than they attribute speakers, even when given the arrow-to-speaker rule. Adding the cue changes candidate scores, and decoding these changes improves speaker attribution by 8.5–15.7 points. Complete three-target operator orbits recover relation-sensitive responses when raw answers remain unchanged, outperforming matched shams on a controlled benchmark and authored comics. A second directed relation reproduces the behavioral gap and cue-specific recovery. Activation replacement finds an early visual effect that fades as an answer-position effect emerges. Under a reversed semantic codebook, predictions remain more closely aligned with the physical operator than with the instructed mapping. These results distinguish reference grounding, sensitivity to the operator, and use of the relation to select an answer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, graphical prompting, relational reasoning, counterfactual decoding, multimodal evaluation, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vQdHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is mainly about whether VLM can locate a graphical cue with the relation that cue expresses.Across 450 paired conditions from 150 scenes, five open VLMs locate arrow endpoints 8.5–13.1 points more accurately than they attribute speakers and stating that "the arrow points to the speaker" does not narrow the gap.The paper give two training-free functions: prompt-effect decoding (PED) and a complete K = 3 operator orbit.These raise relation accuracy by 7.2 to 22.9 points.The complete three-target operator orbit recovers relation-sensitive responses while the raw answers remain

Critical strengths

The experiment in the paper is rigorous enough,which align the models' input para meters and have six renderings.The experiment is sufficiently and convincingly done.

The issue framework in the paper has general-purposing value where reference grounding,sensitivity to the operator and the use of the relation to select an answer significantly contribute to understanding the issue.

Critical weaknesses

The model tested in the paper are all open-source models equal or under 8B ,lacking the test for closed-source or more recent models which is disadvantage to the significance of the paper. Newer models may even have solved this issue.

The paper only phenomenologically describes its findings ,where,"distinguish reference grounding, sensitivity to the operator, and use of the relation to select an answer.“ Adding some analysis of the reason might help readers better understand the mechanistic explanation at the principle level.

The PED function in the paper seems have limited relationship with its finding of the gap between the use of the relation to select an answer and reference grounding and sensitivity to the operator

The paper's logic is not clear enough, and many expressions are hard to understand.Less self-coined terms might be better for readers to follow.

Questions for authors

Can author show the experiment result on models which are more advanced ,newer or merely have more parameters?

Can author explain mechanistic reasoning on why Q3 didn‘t help .If not, hoping for explaining why not investigating further?

Can further explain the relationship between the function mentioned in the paper and the issue and observation that the paper offered?

Reviewer_nkFcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the difference between a VLM’s ability to recognize graphical cues and its ability to utilize the relational meanings of those cues, and introduces the Graphical Prompt Effect to identify relationship-related signals that are not reflected in the final prediction. Using Complete Operator Orbit, the authors demonstrate that by changing the relational operator on the same image—even when the model’s final answer remains unchanged—relationship-sensitive information still exists in the logits and can be recovered via Orbit Decoding. Activation patching further reveals early visual-token and later answer-position effects, while a separate codebook-reversal experiment shows that operator sensitivity does not necessarily imply correct semantic answer binding.

Critical strengths
  1. Rather than relying solely on final prediction accuracy, this paper analyzes logit changes to reveal operator-sensitive signals that may be masked by existing answer priors. In particular, the introduction of PED and Complete Operator Orbit provides a useful perspective for distinguishing candidate-specific preferences from relation-dependent responses, offering a meaningful methodological contribution to understanding how VLMs process graphical prompts.
  2. The paper provides extensive experimental validation beyond simply demonstrating performance improvements from Orbit Decoding. For instance, matched sham controls, rendering variations, and label derangement help examine alternative explanations involving visual artifacts and positional shortcuts. Furthermore, evaluations across different image domains and relational settings, together with statistical significance testing, strengthen the reliability of the main empirical findings.
Critical weaknesses
  1. The experimental validation is still largely limited to arrow- or tail-based directional cues. Although the authors demonstrate generalization across different image domains and relational tasks, these evaluations primarily rely on similar graphical structures. It remains unclear whether the observed operator sensitivity and the effectiveness of Orbit Decoding generalize to other types of graphical cues. Given that the paper aims to investigate relational visual prompts more broadly, it would be valuable to include additional experiments with non-arrow graphical cues, such as connector lines, brackets, or color-coded links. Such experiments would help establish whether the findings reflect a general property of relational visual understanding rather than a phenomenon specific to directional arrow-like cues.
Questions for authors

Q1. Human Screening Protocol. Could the authors clarify how the human screening process described in Section 4 was conducted, including the number of annotators, screening criteria, and how disagreements were resolved?

Q2. Mechanistic Analysis. How specific are the reported activation-patching effects to relational processing? In particular, would similar layerwise patterns emerge from non-relational visual edits, rather than reflecting mechanisms specific to relational understanding?

Q3. Generalization Across Operator Orbit Sizes. The experiments primarily focus on complete operator orbits with K=3. It would be helpful to examine whether the findings generalize to different numbers of relational targets (e.g., K>3) and how orbit size affects the effectiveness of relational signal recovery.

Reviewer_iyfbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether vision-language models can use arrow direction to determine who is speaking. The authors find that although the models produce incorrect answers, the scores of candidate answers may still change with the arrow’s direction; by comparing scores with and without arrows, as well as scores across three different arrow directions in the same scene, some correct answers can be recovered.

Critical strengths
  1. Arrows that do not point to any person, label swapping of persons, and other comparative controls make the results more convincing than simple accuracy comparisons.
  2. The authors compute uncertainty on a per-scene basis and also report cases where performance is insignificant or deteriorates.
Critical weaknesses
  1. The authors should clarify more explicitly which step constitutes the paper’s novel contribution. ORBIT’s approach is to subtract, from each participant’s score for a given answer, the average score that participant assigned across the three downward-pointing arrows, then select the answer with the highest resulting score (pp. 3–4, Eq. 3). Applying the log-score calibration method from [1] to the same three images yields the identical top answer. PED compares scores before and after adding arrows, a procedure closely related to existing visual comparison methods [2].

  2. Current experiments do not sufficiently demonstrate that errors stem from misunderstanding the meaning conveyed by arrows. Although the correct answers to “to whom does the arrow point?” and “who is speaking?” coincide, the question formulations and required commonsense reasoning differ. Another experiment modifies the rule across 20 scenes, requiring the model to answer according to a specified mapping. This change additionally burdens the model with memorizing and executing that mapping (Table 1, Section 6, Table A4).

  3. The computation method for Table 1 requires clarification. Section 4 states that we queried each of the 450 image conditions with five prompts, and determined the final answer per image by majority vote. If the reported accuracy in the table derives from 450 final answers, 93.5% cannot be obtained by rounding to one decimal place: 420 correct answers yield 93.3%, and 421 yield 93.6%. Was accuracy instead computed separately for each of the five prompts and then averaged? Please specify, for each result: the number of correct answers, the denominator, the averaging method, and how ties were resolved, and confirm that confidence intervals follow the same computational protocol.

[1] Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering

[2] Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

[3] Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.