acceptodds
Withdrawn from ICLR 2027

ESCBench: Benchmarking Context-Sensitive Spatial Deixis in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating spatial demonstratives requires asking whether a response to context retains its direction and distance profile across observations of the same situation. Recognition accuracy, prompt stability, and aggregate human resemblance each leave this question unresolved. We introduce ESCBench, a diagnostic benchmark of English this and that that crosses changes in depicted context with changes in observation under a fixed speaker task. Its 576 tabletop scenes vary distance, tool configuration, interlocutor layout, shape, and color, yielding 2,304 images from four viewpoints. Linked comparison identities support tool and layout contrasts within each view; seven protocols connect these contrasts to attribute recognition and elicitation sensitivity. Across 22 vision–language model deployments, the same tool contrast can reverse across views despite high overall recognition and low overall task-form sensitivity. Similar external-human distribution scores can also accompany a large positive tool response or a near-zero one, while pooling views can yield a closer match than every individual view. These findings distinguish contextual responses that agree across observations from observation-dependent responses and aggregate resemblance produced by mixing them. ESCBench makes these distinctions directly testable through matched comparisons.

Primary area:
datasets and benchmarks
Keywords:
spatial deixis, multimodal evaluation, embodied cognition, synthetic datasets, prompt sensitivity, referring expressions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ooT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Building on cognitive psychology studies of spatial demonstratives (Coventry et al., 2008; 2023), ESCBench tests how vision-language models choose between "this" and "that" from a fixed speaker's perspective (MIMo speaking to iCub) in simulated MuJoCo tabletop scenes. The benchmark crosses 12 object distances, 2 tool states (bare hand vs. held rod), 2 interlocutor layouts (side-by-side vs. opposite), 3 shapes, and 4 colors across 4 camera views (speaker eye, addressee eye, external Exo, and top-down Top), yielding 2,304 images. Paired with seven prompting protocols across 22 models, the evaluation shows that strong object recognition and low prompt sensitivity do not guarantee viewpoint invariance, and that aggregate distributional similarity (JSD) to external human studies can mask opposing viewpoint biases or flat outputs.

Critical strengths

Crossing scene factors with camera views while holding the designated speaker fixed clearly exposes why single-view evaluations and aggregate distribution matching (such as JSD against human studies) can be misleading. Even a model with 100% object recognition and low prompt-format sensitivity like Gemini 3.1 Pro flips the sign of its tool-use effect at M04 from +12.5 pp in Exo to -33.3 pp in Top.

The evaluation across 22 models carefully separates net directional shifts from total choice switching and pairs task-wording changes with option-order reversals. Showing that pooling views gives Qwen3.8-27B a lower JSD (0.133) than any single view on its own (0.231 to 0.400), and that near-zero net contrasts can hide nearly 50% two-way switching, is a valuable diagnostic lesson.

Critical weaknesses
  1. The benchmark has no same-image human annotations on its rendered MuJoCo scenes, relying instead on human distributions from physical memory-game experiments (Coventry et al., 2008; 2023). Without human judgments on these actual 2D renders (such as the orthographic Top view where spheres and cylinders look like identical circles), it is hard to separate VLM reasoning failures from visual ambiguity, especially since random guessing achieves a better external JSD (0.276) in Table 1 than Gemini 3.1 Pro (0.304) and GPT-6 Sol (0.295).

  2. Unless I am misreading Section 3.3, the held rod physically touches the target at M03 to M05 while the bare hand points without contact, and model tool effects appear almost entirely in M01 to M05 before vanishing at M06 to M12 (Figures 3a and 5). This suggests models may be reacting to 2D visual contact rather than tool-extended reachability, while differing Ego camera gaze protocols across tool states (Section 3.2) mix camera shifts into the participant-view tool contrasts.

  3. The manuscript prose reads like a dense technical audit log packed with repetitive caveats, and the 62-page PDF includes 16 pages of appendices (Appendices N to S) on 50-decimal interval-arithmetic proofs, PD controller gains, and nanometre cloth clearances for historical 7.5 mm disk tasks that are not part of the VLM benchmark.

Questions for authors
  1. Maybe I am misinterpreting the contact states at M03 to M05, so I would appreciate the authors' perspective: since the rod touches the target at M03 to M05 while the bare hand does not, how can we tell if models respond to tool-extended reachability rather than a 2D visual cue of physical contact (and why do the Ego camera gaze protocols differ between bare-hand and held-rod scenes)?

  2. For the five non-deterministic API deployments, how does the test-retest flip rate on identical prompts compare to the reported 5% to 22% task and order flip rates, and could the unused historical disk appendices (Appendices N to S) be removed?

Reviewer_PL7MOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ESCBench studies whether English demonstrative choices respond consistently to depicted context across observation views. It crosses distance, tool configuration, interlocutor layout, shape, and color in 576 scenes, yielding 2,304 images. Seven protocols evaluate 22 model deployments through matched contextual contrasts, attribute recognition, prompt sensitivity, and comparisons with external aggregate human distributions. The main finding is that accurate recognition, stable outputs, and pooled human resemblance can coexist with inconsistent conditional responses. The contribution is a controlled descriptive benchmark, rather than a per-image test of human-correct demonstrative use.

Critical strengths
  1. Matched comparisons reveal behavior hidden by aggregate scores. The benchmark separates response direction, distance profile, view dependence, and elicitation sensitivity. For example, Gemini's M04 held-minus-bare contrast changes from +12.50 percentage points in Exo to -33.33 in Top despite perfect overall attribute recognition (Figure 3a). Qwen3.8-27B's pooled-view human-reference JSD is 0.133, below every individual-view value of 0.231–0.400 (Figure 4e). These examples substantiate the central contribution: a favorable marginal summary can fail to describe any particular observation or contextual contrast.

  2. Careful measurement and interpretation. The analysis retains invalid responses and paired-transition denominators, distinguishes constant-output stability from contextual responsiveness, and excludes projection-ambiguous Top sphere/cylinder cases from primary shape scoring. The human comparisons use condition-wise square-root Jensen–Shannon distance and are explicitly descriptive. The linked scene/template/family organization and disclosed deployment limitations make the results more interpretable than a single composite leaderboard.

Critical weaknesses
  1. The strongest localized contrasts need a repeatability check. Figure 3a's M04 example uses only 24 matched pairs per view, with one recorded response per input. Appendix A acknowledges that API outputs may vary and that stored-response reanalysis does not repeat inference. Thus, the observed reversal is established for these records, but its stability under the same deployment is unknown. This matters because the reversal is a central example of observation-dependent contextual behavior. Separately, four early open-weight deployments have configuration differences in some prompt comparisons; these qualifications should remain attached to the relevant results rather than being generalized to all deployments. A small repeated evaluation of the headline contrasts would substantially strengthen the empirical case.

  2. The benchmark diagnoses response dependence more directly than its source. Correct color and shape do not establish correct visual grounding of the speaker, tool contact, or target distance. D2 supplies participant roles in the prompt but still requests only color and shape (Appendix D.5), so it does not independently measure role grounding. Even camera-matched Exo/Top tool pairs change posture, contact, and occlusion. The paper explicitly acknowledges these factors and avoids claiming an isolated affordance effect; accordingly, this is a limitation on diagnostic specificity rather than a contradiction of its descriptive results. A focused prerequisite-perception check would clarify when the disagreement reflects deictic interpretation rather than changed access to relevant visual evidence.

Questions for authors
  1. Can the highlighted Gemini M04 Exo/Top contrasts be repeated several times under a fixed deployment configuration? Please report response variation for identical inputs and whether the signs remain stable. A small targeted repeat is sufficient; a rerun of the full benchmark is unnecessary.

  2. For the camera-matched cases underlying the key contrasts, can models correctly locate the designated speaker and identify tool contact and coarse target distance? How much disagreement remains among cases passing those checks? This would directly refine the interpretation of the benchmark's failures.

  3. Which highlighted comparisons were selected using the reused 192-image development panel, and where can the anonymous package linking raw responses, scene families, prompts, and execution configurations be accessed? The paper documents these dependencies; making the corresponding records accessible would allow the principal descriptive findings to be checked.

Reviewer_zYv5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ESCBench is introduced to evaluate spatial demonstratives across observations of the same situation. The diagnosis includes confounders such as recognition accuracy and prompt stability. The evaluation results separate contextual responses from observation-dependent responses and aggregate resemblance produced by mixing them.

Critical strengths

The major strength of this paper is a comprehensive and detailed benchmark taxonomy, with how each observation/prompt variation is controlled.

Critical weaknesses
  1. The clarity of the writing makes the paper hard to follow. For example, the abstract uses undefined concepts "fixed speaker task", "prompt stability", " Linked comparison identities support tool," and "human resemblance," which can only be decoded after reading the main paper as context. Concepts such as "elicitation sensitivity" and "prompt stability" are used interchangeably without proper clarification. The presentation obscures the main contribution, and the paper would benefit from substantial improvements in clarity.
  2. The proposed benchmark has limited task realism, diversity, and evaluation justification. The paper does not provide sufficient motivation for its task design and evaluation protocol. In particular, the fixed-speaker setting represents a simplified form of spatial communication that overlooks important contextual factors, such as gaze, attention, and broader conversational context. Furthermore, the benchmark relies on synthetic scenes with limited diversity, raising concerns about its validity and generalizability to real-world scenarios. Finally, the rationale for evaluating response consistency across different viewpoints and tool configurations is not sufficiently established. Changes in spatial context may legitimately affect demonstrative choices. A clearer justification of these evaluation assumptions is needed to establish the benchmark's validity and practical relevance.
Questions for authors
  1. Could you please clarify how human responses are obtained and how to verify the quality of human responses on the proposed synthetic dataset?
  2. Could you address the second weakness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.