- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Building on cognitive psychology studies of spatial demonstratives (Coventry et al., 2008; 2023), ESCBench tests how vision-language models choose between "this" and "that" from a fixed speaker's perspective (MIMo speaking to iCub) in simulated MuJoCo tabletop scenes. The benchmark crosses 12 object distances, 2 tool states (bare hand vs. held rod), 2 interlocutor layouts (side-by-side vs. opposite), 3 shapes, and 4 colors across 4 camera views (speaker eye, addressee eye, external Exo, and top-down Top), yielding 2,304 images. Paired with seven prompting protocols across 22 models, the evaluation shows that strong object recognition and low prompt sensitivity do not guarantee viewpoint invariance, and that aggregate distributional similarity (JSD) to external human studies can mask opposing viewpoint biases or flat outputs.
Critical strengths
Crossing scene factors with camera views while holding the designated speaker fixed clearly exposes why single-view evaluations and aggregate distribution matching (such as JSD against human studies) can be misleading. Even a model with 100% object recognition and low prompt-format sensitivity like Gemini 3.1 Pro flips the sign of its tool-use effect at M04 from +12.5 pp in Exo to -33.3 pp in Top.
The evaluation across 22 models carefully separates net directional shifts from total choice switching and pairs task-wording changes with option-order reversals. Showing that pooling views gives Qwen3.8-27B a lower JSD (0.133) than any single view on its own (0.231 to 0.400), and that near-zero net contrasts can hide nearly 50% two-way switching, is a valuable diagnostic lesson.
Critical weaknesses
-
The benchmark has no same-image human annotations on its rendered MuJoCo scenes, relying instead on human distributions from physical memory-game experiments (Coventry et al., 2008; 2023). Without human judgments on these actual 2D renders (such as the orthographic Top view where spheres and cylinders look like identical circles), it is hard to separate VLM reasoning failures from visual ambiguity, especially since random guessing achieves a better external JSD (0.276) in Table 1 than Gemini 3.1 Pro (0.304) and GPT-6 Sol (0.295).
-
Unless I am misreading Section 3.3, the held rod physically touches the target at M03 to M05 while the bare hand points without contact, and model tool effects appear almost entirely in M01 to M05 before vanishing at M06 to M12 (Figures 3a and 5). This suggests models may be reacting to 2D visual contact rather than tool-extended reachability, while differing Ego camera gaze protocols across tool states (Section 3.2) mix camera shifts into the participant-view tool contrasts.
-
The manuscript prose reads like a dense technical audit log packed with repetitive caveats, and the 62-page PDF includes 16 pages of appendices (Appendices N to S) on 50-decimal interval-arithmetic proofs, PD controller gains, and nanometre cloth clearances for historical 7.5 mm disk tasks that are not part of the VLM benchmark.
Questions for authors
-
Maybe I am misinterpreting the contact states at M03 to M05, so I would appreciate the authors' perspective: since the rod touches the target at M03 to M05 while the bare hand does not, how can we tell if models respond to tool-extended reachability rather than a 2D visual cue of physical contact (and why do the Ego camera gaze protocols differ between bare-hand and held-rod scenes)?
-
For the five non-deterministic API deployments, how does the test-retest flip rate on identical prompts compare to the reported 5% to 22% task and order flip rates, and could the unused historical disk appendices (Appendices N to S) be removed?