acceptodds
Withdrawn from ICLR 2027

GEOMETRIC PARROTS: RULE ABSENCE VS. RULE FRAGILITY IN SPATIAL AND PHYSICAL REASONING FOR VISION- LANGUAGE MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models have been characterised as stochastic parrots- fluent producers of plausible text without grounded understanding of what that text describes. We ask whether the same holds for vision-language models, and observe that accuracy cannot answer the question. A model that answers a geometric question correctly and then abandons that answer the moment a user asserts otherwise did not hold the rule; a model that answers wrongly and revises toward the truth is reasoning well. Accuracy scores these identically, because it evaluates each response in isolation. What separates them is the direction in which an answer moves under challenge, and direction is egible only where the correct answer is known exactly — which annotated benchmarks, whose labels are human judgements about pre-existing images, cannot provide. We present GRIP, a suite of 34 procedurally generated domains comprising 100,000 images and 600,000 questions spanning plane, solid, projective, transformational, topological, analytic, inductive, optical and physical reasoning. Every scene is drawn by deterministic code and every answer computed in closed form from the coordinates the renderer itself uses; ground truth is derived rather than annotated, and no generative model participates at any stage. On this foundation we introduce Dual-Loop Evaluation. The closed loop poses five ordered questions per image on a compositional ladder in which each level consumes the previous level's result, localising where competence stops. The open loop then challenges a free-response answer twice over: under cross-examination, where the model is shown the answers other models gave, and under answer sycophancy, where a user asserts an correct answer without evidence. Because ground truth is exact, every resulting change of answer is signed as a correction or a capitulation — a decomposition that existing work on answer instability reports only as an unsigned flip rate. Rule absence and rule fragility thus become separately measurable: the first as failure in the closed loop, the second as a correct answer that does not survive challenge.

Primary area:
datasets and benchmarks
Keywords:
Geometric Understanding, Physical Understanding, Spatial Intelligence, Robotics, Benchmarks, Multi-modals, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KsDgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GRIP, a procedurally generated benchmark for geometric, spatial, and physical reasoning. It evaluates both initial answers and responses to user challenges or peer answers, measuring when models correct mistakes and when they abandon correct answers.

Critical strengths

The motivation is relevant, as single-turn accuracy does not fully capture reliability during interaction. Procedural generation provides control over scenes and reference answers. The no-challenge re-asking control is also useful for distinguishing challenge-induced changes from ordinary answer variability.

Critical weaknesses
  1. The contribution over prior work is unclear. PENDULUM already distinguishes incorrect-to-correct and correct-to-incorrect transitions through progressive and regressive sycophancy. Although the protocols differ, the central distinction is established. Procedural generation is useful, but the paper does not clearly show what additional diagnostic insight GRIP provides beyond its construction and task coverage.

  2. The scoring protocol weakens the main comparisons. Appendix D.3 and Table 10 state that open-weight and hosted models use different comparators and that their scores are not directly comparable. However, Table 3 and Figure 3 present these groups together, and the paper discusses their performance separation. A shared scoring procedure is needed to support these conclusions. Appendix B.6 also lacks sufficient detail about the semantic validator and its human verification, including audit size and agreement rates.

Questions for authors

What does GRIP reveal that cannot be captured by the corrective and regressive behaviors already studied in PENDULUM?

Can the authors report results using a shared scoring procedure and provide quantitative details of the human audit?

Reviewer_GgWAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a procedurally generated benchmark for 34 domains and evaluates VLMs' reasoning accuracy and answer stability under user assertions and peer challenges.

Critical strengths

The benchmark covers a relatively broad range of geometric and physical reasoning tasks, with programmatically generated reference answers that could facilitate scalable and controlled evaluation.

Critical weaknesses

Incomplete exposition and poor organization:

  • The central distinction between "rule absence" and "rule fragility" is not sufficiently justified. An initially correct answer does not necessarily indicate that the model has acquired the underlying reasoning rule, and changing a correct answer under external pressure does not establish that the rule itself is fragile.
  • The Introduction is unusually short and incomplete.
  • Section 3, which introduces the main contribution of the paper, is severely underdeveloped.
  • The Related Work section is too brief and misses several directly relevant studies. The bibliography contains too few references.
Questions for authors

Some problems with exposition and organization:

  • Section 3 is severely underdeveloped. The authors provide almost no concrete details about individual task construction, scene generation, question generation, or difficulty control. There are no qualitative examples either. Simply stating that the scenes and answers are generated deterministically does not constitute an adequate methodological description for a benchmark. Could the authors provide more details?
  • The paper does not adequately introduce its evaluation settings. The labels E1, E2, E3, and E4 are used throughout the main text without a unified explanation of the four experiments. Besides, the organization of Section 5 is clearly incorrect. Section 5.1 is titled "Grain Robustness (E2)," yet Section 5.1.1 is "Answer Sycophancy (E3)" and Section 5.1.2 is "Open-ended Cross-examination (E4)." How do the authors organize their experiments?
  • Important quantitative results are not presented where they are discussed. For example, Section 5 emphasizes that baseline-adjusted accuracy reveals substantially larger differences between models, yet the main results table reports only raw accuracy.
  • Appendix D.3 explicitly states that the hosted and open-weight models were evaluated using different comparators and that their scores are not directly comparable. Why does the main text nevertheless discuss performance differences between these two groups?
Reviewer_uy1EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the concern that vision-language models may change correct answers when challenged externally during geometric, spatial, and physical reasoning, the paper introduces the GRIP benchmark and a dual-loop evaluation framework. It programmatically generates 100,000 images and 600,000 questions across 34 domains, using exact ground-truth answers to assess both reasoning accuracy and stability under external interference. The framework also distinguishes beneficial correction from erroneous concession. Experiments with 16 models reveal a clear gap between accuracy and stability, suggesting that conventional single-round accuracy alone cannot fully capture reasoning reliability.

Critical strengths
  1. The paper is well structured and focuses on both the accuracy and stability of vision-language model reasoning, making the investigation practically relevant.
  2. GRIP covers a broad range of reasoning domains, and the extensive experiments systematically examine different models’ performance and their stability under external interference.
Critical weaknesses
  1. The related-work discussion is limited. The study spans geometric and physical reasoning in vision-language models, reasoning stability, and interactive evaluation, yet it cites only about 21 references. The Related Work section is also relatively brief and does not sufficiently cover or discuss existing research in these areas.
  2. Figure 2 does not adequately illustrate the overall evaluation pipeline. It presents only high-level stages, such as data generation, model execution, and result scoring, and provides too little detail to make the process intuitive.
Questions for authors
  1. Could the discussion of related work be expanded, particularly by comparing GRIP with existing visual reasoning benchmarks and studies of multimodal reasoning stability, to better establish its novelty and incremental contributions?
  2. Could the inconsistencies across experimental descriptions be clarified? For example, pages 8 and 31 state that Natural Peers and Constructed Consensus in Experiment E4 use the same test samples and denominator, allowing direct comparison. However, Table 15 on page 33 reports substantially different sample counts for the two conditions. For GPT-5.6 Sol, the totals are 1,724 and 2,000, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.