acceptodds
Under review as a conference paper at ICLR 2027

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

Abstract

As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding? We identify a pervasive vulnerability, the Cartesian Shortcut: models frequently discretize the orthogonal grid-based layouts prevalent in visual reasoning benchmarks into explicit textual coordinates, offloading reasoning from visual perception to text-based deduction. To re-evaluate visual reasoning when this shortcut is unavailable, we introduce Polaris-Bench, which re-formulates 53 visual reasoning tasks in Polar coordinate space with paired Cartesian counterparts that preserve task semantics, disrupting the orthogonal structure that models exploit. Comprehensive evaluation across state-of-the-art MLLMs reveals that frontier models achieving – on Cartesian layouts collapse to – on Polar equivalents. Moreover, thinking gains largely vanish on Polar layouts, prompting interventions fail to close the gap, and comparable drops arise on other non-orthogonal layouts. These findings reveal that current MLLMs' visual reasoning performance is strongly coupled to orthogonal grid structure, a fragility that is consistent across model families but far smaller in humans, who maintain 88.8% accuracy on Polar layouts.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Vision Reasoning Benchmark, Evaluation, Cartesian Shortcut, Coordinate Space

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_zKVMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This article proposes Polaris-Bench, which divides 53 types of visual reasoning tasks into two versions, grid and polar coordinates, to test the model's adaptability to layout changes.
  • Several strong models dropped significantly on the polar coordinate version, and increasing thinking and providing examples failed to effectively bridge the gap. The authors believe that existing models rely too much on regular grids and their textual coordinate representation.
Critical strengths
  • The starting point of the question is interesting. If you change the layout of the same question, you will obviously lose points, which shows that the conventional visual reasoning assessment may indeed miss important issues.
  • A paired experimental design is better. The author distinguishes tasks with complete logical alignment and topological changes, reducing the interference of "actually the topic has changed".
  • The experiment covers a wide range of areas. Containing multiple strong models, inference patterns, and other geometric deformations, the observed phenomena are not isolated cases of a single model or torus layout.
Critical weaknesses
  • Perception and reasoning are not sufficiently separated. Correctly identifying the grid size does not mean correctly reading the grid content and adjacency relationships. Therefore, the existing experiments are insufficient to illustrate how problematic reasoning itself remains after perceptual errors are ruled out.
  • The influence of training distribution is not excluded. The model may simply have seen less polar tasks. In the absence of training or adaptation experiments, it is unclear whether this is a shortcoming that can be replenished through data, or a deeper capability limitation.
  • The actual mission significance is unclear. The experiment mainly focuses on artificially generated grid questions. Whether it can reflect the problems in real charts, maps or other visual scenes still needs to be verified.
Questions for authors
  • Can we provide the correct lattice content and adjacency relationships, and further measure how many inference errors are left in the model after excluding visual reading errors?
  • How much could the performance gap be reduced by adding a small amount of polar training data, or adding explicit numbers to rings and sectors?
  • Can a similar phenomenon be verified on real graphs or maps, showing that this finding is not limited to synthetic grid tasks?
5 comments
AuthorsOpenReview ↗

Response to Reviewer zKVM

We thank Reviewer zKVM for noting that conventional visual reasoning evaluation may miss important issues, and for recognizing our paired design and broad coverage across models and layouts.

 

Q1 & W1: Perception vs. Reasoning Separation, the Two-Stage Analysis, and Ground-Truth Textual Representation

A: Please see our Overall Response (Scope and Mechanistic Attribution and Two-Stage Experiment) for a unified response. Specifically, our paired/image-based Stage-2 control and the 983-pair analysis (Overall Response, Table) show that the ~40pp Cartesian–Polar gap holds when caption bottlenecks and pairing bias are excluded. Regarding the pure-text setting, as discussed in Overall Response (d), fully expressing an image as structured text would allow the model to bypass visual processing entirely and leverage text-based matrix reasoning regardless of the original layout. The explicit-coordinate experiments on Polar layouts (P+axis and P+cell, Q2 below) also partially address this question.

AuthorsOpenReview ↗

Response to Reviewer zKVM [part 2]

Q2 & W2: Training Distribution and Explicit Ring/Sector Numbers on Polar Layouts 

A: Regarding training: We share the reviewer’s view that training-level interventions are a valuable direction to further explore, and to help disentangle training-distribution familiarity from deeper capability limitations, which aligns with paper Section 7 ("separating the contributions of familiarity, geometry, and orthogonal structure through training-level interventions ... remain important open directions"). Consistent with our Overall Response, this paper focuses on identifying the orthogonal-grid evaluation confound, constructing Polaris-Bench, and empirically characterizing layout sensitivity across off-the-shelf MLLMs, conclusions that hold without claiming a deeper capability limitation. Parameter-level training is therefore outside the scope of this diagnostic study and remains a promising future direction that Polaris-Bench is designed to facilitate as a testbed.

Regarding adaptation experiments: On the test-time adaptation side, we evaluated 5-shot in-context learning with solved Polar examples and explicit Cartesian-conversion hints across six models and six tasks (Section 5, Figure 7(f), and Appendix D.3 Table 20), finding that neither recovers Polar accuracy (changes within <4 pp for hints, and lower accuracy on five of six models for 5-shot ICL).

Regarding explicit ring/sector numbers on Polar: Following Reviewer zKVM’s suggestion (combined with coordinate-perturbation suggestions from other reviewers), we conducted a paired experiment on five tasks (Template Matching, Grid Folding, Largest Number Path, Lattice Paths, and Path Counting) across six representative models:

  • P: original Polar layout;
  • P+axis: labeling explicit ring and sector numbers along the Polar axes;
  • P+cell: labeling explicit (ring, sector) coordinates inside each Polar cell;
  • C: original Cartesian layout;
  • C+WD: stating wrong grid dimensions in the prompt;
  • C+WC: labeling shuffled wrong (row, col) coordinates in every cell;
  • C+SC: repeating the same random wrong coordinate in every cell as a control.

All conditions use the exact same paired instances (100 pairs per task) and a unified, non-occluding label style. The table below reports the five-task mean accuracy (%):

ModelCC+WDC+WCC+SCPP+axisP+cell
Gemini-3.1-Pro87.684.462.681.435.738.253.8
Gemini-3-Flash76.464.055.868.236.435.643.4
GPT-5.287.468.057.676.435.638.447.0
Claude-Sonnet-4.642.734.034.233.625.422.023.6
Gemma-4-31B67.863.252.465.229.233.835.2
Kimi-k2.566.143.051.656.429.129.831.4
Average71.359.452.463.531.933.039.1
  • On Polar layouts, relative to the 39.4 pp six-model average C–P gap (C: 71.3% vs. P: 31.9%), P+axis remains close to P (33.0%, narrowing the gap by 1.1 pp), while P+cell raises the six-model average to 39.1% (narrowing the gap by 7.2 pp overall: +18.1 pp on Gemini-3.1-Pro and +11.4 pp on GPT-5.2, vs. −1.8 to +7.0 pp on the other four models). The gains on stronger models under P+cell are directionally consistent with our observations on Monotonic Path (Section 5) and the digit-labeled pilot version of Four Color (Appendix B.4) that in-cell numerical anchors can aid text-based tracking on Polar layouts, though unlike task-intrinsic cell values in those two tasks, generic coordinate labels here recover only part of the Cartesian-Polar gap.
  • On Cartesian layouts, all six models drop in accuracy under all three perturbations relative to C (averaging 63.5% on C+SC, 59.4% on C+WD, and 52.4% on C+WC, down from 71.3%), and shuffled wrong coordinates (C+WC) cause a larger drop than the repeated-coordinate control (C+SC) in five of the six models.
  • Overall, on these five tasks, perturbing explicit coordinates lowers Cartesian accuracy, whereas providing ground-truth coordinates on Polar layouts improves performance for some models (particularly stronger ones), though a substantial Cartesian-Polar gap remains across all models.

 

AuthorsOpenReview ↗

Response to Reviewer zKVM [part 3]

Q3 & W3: Practical Significance and Connection to Real-World Graphs, Maps, and Visual Scenes

A: We deliberately adopt procedural generation to construct a controlled paired testbed, which is essential for isolating coordinate-layout effects under aligned task rules, a level of control that cannot be achieved with naturally occurring images. Because such pair-wise data is difficult to collect from naturally occurring images, we use procedural synthesis to build this controlled comparison.

Connection to real-world scene: While Polaris-Bench is constructed in a synthetic environment, the layout sensitivity it reveals has direct implications for both benchmark evaluation and real-world applications. 

  • First, current visual reasoning benchmarks rely heavily on orthogonal grid layouts (Section 2: over 56% of examples across 9 benchmarks elicit explicit textual coordinate discretization), indicating that existing evaluations may overestimate model robustness across broader visual layouts. 
  • Second, as the reviewer noted, many real-world visual scenarios naturally lack orthogonal grid structure, such as analog clocks and dials, circular or radar charts, and transit or geographic maps. Recent empirical studies and benchmarks on real-world scenarios report observations consistent with our Cartesian–Polar findings, showing that frontier MLLMs also struggle on real-world tasks with non-orthogonal or complex spatial layouts, including analog clock reading (e.g., Lost in Time (Saxena et al, 2025), pointer meter and dial reading (e.g., DialBench, Wang et al., 2025), non-orthogonal chart understanding such as radar and rose charts (e.g., ChartX, Xia et al., 2024), and route planning or spatial reasoning on real transit and cartographic maps (e.g., FRIEDA, Pyo et al., 2026). 

Accordingly, our controlled Cartesian–Polar testbed provides a diagnostic reference for evaluating layout robustness, and serves as a foundation for future work to connect controlled layout transformations with real-world visual scenarios and study the underlying mechanisms.

 

References:

  • Saxena, Rohit, Aryo Pradipta Gema, and Pasquale Minervini. "Lost in time: Clock and calendar understanding challenges in multimodal LLMs." arXiv preprint arXiv:2502.05092 (2025).
  • Wang, Futian, et al. "DialBench: Towards Accurate Reading Recognition of Pointer Meter using Large Foundation Models." arXiv preprint arXiv:2511.21982 (2025).
  • Xia, Renqiu, et al. "Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning." IEEE Transactions on Image Processing (2025).
  • Pyo, Jiyoon, et al. "FRIEDA: Benchmarking multi-step cartographic reasoning in vision-language models." International Conference on Learning Representations. Vol. 2026. 2026.
Reviewer_6yAKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Polaris-Bench, containing 5300 cartesian-polar pairs across 53 visual reasoning tasks. Evaluated on 14 multimodal models, the authors find large accuracy drops on Polar layouts and generally smaller gains from extra reasoning. The authors also experiment with other geometric deformations and prompting interventions which further valid this sensitivity. These findings are interpreted as evidence that models rely on familiar orthogonal grids to convert visual problems into textual representations.

Critical strengths
  1. I liked the decision to separate tasks with identical rules and answers from those affected by changes in boundary conditions. The performance gap remains large in the fully aligned group, so the result cannot simply be explained by the Polar versions being different problems. The task construction and validation described in Appendix B also show considerable care.

  2. The accuracy drops are large enough to be convincing and appear across several model families. For example, Gemini-3.1-Pro drops from 82.6% to 35.9%, and GPT-5.2 from 77.4% to 39.2%. The wave and arc experiments are especially helpful: they show that the problem extends beyond the particular features of Polar layouts.

Critical weaknesses
  1. My main concern is the explanation of these results. The experiments clearly show that models struggle when the layout changes, but they do not establish that using textual coordinates is the cause. Writing down coordinates can be a reasonable way to solve a visual problem. The caption experiment does not settle this either. The paper would be stronger if it kept the demonstrated layout sensitivity separate from the proposed shortcut explanation.

  2. I am also unsure how to interpret the human baseline. Appendix B.5 says that submissions with incomplete or insufficiently justified reasoning were returned for revision until accepted. The reported scores use these accepted submissions, and the reported times exclude revision cycles. This is useful for checking the dataset, but the human–model comparison depends on what feedback people received and whether they changed their answers. These details need to be clear before drawing conclusions from the much smaller human performance drop.

Questions for authors
  1. Did you check any caption content beyond grid dimensions? Also, were the conditional Cartesian and Polar results computed on the same paired questions, and how many examples remained in each condition?

  2. What feedback did human raters receive during revision? Could they change their answers, and did quality checkers know the ground truth? Are first-submission accuracy results available?

  3. Appendix B.4 describes changes made after pilot evaluations, including replacing numbers with colors in Four Color. Which models informed these decisions, and were the final test instances kept separate from the pilot examples?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6yAK

We thank Reviewer 6yAK for highlighting our careful paired task construction, the convincing accuracy drops across model families, and the evidence that the problem extends beyond Polar layouts.

 

Q1 & W1: Separation of Layout Sensitivity from the Shortcut Explanation, and Two-Stage Experiment Details

A: Please see our Overall Response for our 2-stage experiment explanation and scope clarification. Regarding the specific details of the two-stage experiment: 

  • In the experiment, we only checked grid dimensions against ground-truth metadata and did not verify caption content beyond grid dimensions (as detailed in Overall Response) since this is a valid and reasonable metric that can be checked against metadata uniformly across all 53 tasks. 
  • The conditional results in Figure 1 were computed separately on each layout's Stage-1 filtered subset: with 4,251 examples remaining on Cartesian (5,300×80.2%) and 1,102 examples remaining on Polar (5,300×20.8%), rather than on the same paired questions. 
  • Following the reviewer's question, we further evaluated the 983 paired questions where grid dimensions were correctly extracted in both layouts, and also re-ran the Stage-2 subsets directly using the original images alongside captions (see the table in our Overall Response); we found closely matched accuracy across all settings (88.2% vs. 48.6% with captions and 90.2% vs. 46.6% with original images on the paired subset), with the ~40-point Cartesian–Polar gap remaining consistent.

 


Q2&W2: Human Annotation and Quality-Check Details

A: Quality checkers did not have access to the ground-truth answers and only saw the task instructions and questions (same as the raters) along with the rater's submission. Because this followed the annotation platform's standard paired QA workflow, we did not apply task-specific training or additional constraints to quality checkers. As reviewer suggested, we fetch and report the human annotation detail as below, and also will update to the paper Appendix. 

Across all annotated questions, the first-pass acceptance rate without revision is 80.6% (79.6% on Cartesian and 81.5% on Polar), while 19.4% underwent revision. Because quality checking is an internal platform operation, individual rejection comments are not visible to us, so we analyzed the pre- and post-revision submissions directly (raters were allowed to modify their answers during revision). Among the revised items, 82.0% kept the original answer unchanged and returned a more detailed reasoning write-up (consistent with our requirements of detailed thinking and solution trace); in the remaining 18.0% where the answer changed (16.2% of Cartesian revisions and 19.9% of Polar revisions), this accounts for only 3.5% of all annotated questions overall (3.3% on Cartesian and 3.7% on Polar), with shifts occurring in various directions across options.

On the full set of first-round submissions before quality checking, human accuracy is 92.5% on Cartesian and 86.9% on Polar (−5.6 pp), compared with the final reported accuracy of 94.5% on Cartesian and 88.8% on Polar (−5.7 pp). This confirms that quality-check revisions do not affect the human comparison across layouts, as the small Cartesian-to-Polar gap already holds on first submissions. 

AuthorsOpenReview ↗

Response to Reviewer 6yAK [part 2]

Q3: Appendix B.4 describes changes made after pilot evaluations, including replacing numbers with colors in Four Color. Which models informed these decisions, and were the final test instances kept separate from the pilot examples?

A: Pilot evaluations during task development primarily used Gemini-3-Flash, Gemini-3.1-Pro, and Claude-Sonnet-4.6, with occasional checks on GPT-5.2 on a few tasks. The remaining 10 evaluated models were not involved in task development and exhibit the same performance pattern in our final evaluation (Table 1). All final test instances were generated in a single batch after all task development was completed and the code was frozen, completely separate from all pilot examples. 

More details on the Four Color task design: as stated in Appendix B.4, during early task design we consistently observed large Cartesian–Polar gaps across most tasks, while Monotonic Path and Four Color with digit labels remained robust on both layouts. To form a direct comparison, we kept Monotonic Path unchanged with its text anchors and modified Four Color to be color-based. After this change, Four Color no longer maintained the high Polar performance of its digit-based version. We transparently documented this design evolution and share this informative contrast/finding (Section 5, Appendix B.4). 

Reviewer_2QCDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The Cartesian Shortcut investigates whether strong visual reasoning scores depend on orthogonal grids that models can readily transcribe into textual coordinates. The paper introduces Polaris-Bench, comprising 53 paired Cartesian and Polar visual reasoning tasks, and evaluates 14 multimodal models. Frontier models' accuracy falls from 69–83% on Cartesian layouts to 31–39% on Polar layouts, with substantial drops persisting on logically equivalent tasks. Increased reasoning and prompting interventions do not close the gap, while humans show a much smaller decline. The study provides a useful diagnosis of layout-dependent performance, although the contribution of the proposed shortcut remains insufficiently separated from perceptual difficulty and unfamiliarity.

Critical strengths

The paper identifies a meaningful limitation in how existing benchmarks assess visual reasoning. Many visual reasoning tasks, such as Sudoku and grid navigation, use orthogonal layouts that can be readily transcribed into row-column coordinates or discrete matrices. This allows models to shift much of the problem-solving burden to text-based deduction, making high benchmark scores insufficient evidence of robust visual reasoning across different spatial presentations. The Cartesian Shortcut provides a concrete, testable account of this evaluation concern (Section 1). Polaris-Bench operationalizes it through paired Cartesian and Polar tasks, with performance degradation persisting even when rules, constraints, and ground truths remain identical. This is a valuable diagnostic perspective: it examines whether apparent visual reasoning competence depends on a convenient representational structure and motivates evaluating generalization beyond familiar orthogonal grids.

Critical weaknesses

The proposed Cartesian Shortcut mechanism is not sufficiently separated from perceptual difficulty and unfamiliarity. The paired evaluations convincingly demonstrate sensitivity to visual layout, including on tasks with identical rules, constraints, and ground truths. However, explicit coordinate use in reasoning traces can reflect a legitimate abstraction of correctly perceived visual information; its prevalence alone does not establish that models bypass meaningful visual reasoning (Section 2). Polar conversion simultaneously changes curvature, cell sizes, visual grouping, and familiarity, so the resulting performance gap does not isolate dependence on textual coordinate discretization from these other sources of difficulty. The two-stage experiment provides limited separation: its subset of supposedly correctly perceived instances is selected through grid-dimension accuracy, which does not verify cell contents or their spatial assignments (Appendix C.2). Consequently, the residual task-accuracy gap cannot confidently be attributed to downstream reasoning rather than incomplete or incorrect visual extraction. Although the conclusion acknowledges unresolved contributions from familiarity and geometry, the mechanistic interpretation remains less firmly established than the observed layout sensitivity. Verified text-only baselines, complete transcription checks, and controlled spacing or aspect-ratio manipulations would help distinguish these explanations.

Questions for authors
  • Can the authors compare the image-based evaluation with a matched, purely text-based setting? For logically equivalent tasks that admit an exact symbolic representation, such as Sudoku, could the authors provide the same instances as ground-truth textual inputs without images: (1) a row-column matrix or coordinate-value list and (2) an equivalent ring-sector coordinate-value list, with the corresponding rules stated explicitly? Keeping clues, constraints, target cells, answer options, and inference settings matched would help determine whether the Cartesian-Polar gap also appears when visual extraction is bypassed. Strong performance in both text-only conditions would implicate extracting or using structure from the images, while a remaining gap would suggest that interpreting the ring-sector representation or its rules also contributes. The existing two-stage experiment uses model-generated captions and checks grid dimensions as a coarse proxy for perceptual accuracy (Appendix C.2); it does not ensure that all cell contents and their assignments are correct. An oracle transcription baseline would therefore provide a more direct control for this question, although it would not by itself establish that textual coordinate use causally explains the image-based gap.

  • Do models depend on orthogonal row-column structure, or also on uniform spacing and familiar cell proportions? The geometric deformation experiments examine wave and arc transformations (Section 5; Appendix D, Table 21), but do not systematically isolate cell aspect ratio or row-column spacing while preserving straight, orthogonal grid lines. Could the authors evaluate matched instances with (1) rectangular cells obtained by stretching one axis and (2) nonuniform row heights and column widths, keeping cell contents, logical adjacency, task rules, and ground truths fixed on suitable tasks? Both conditions retain explicit row-column indexing, so the proposed Cartesian Shortcut should remain available even though the visual appearance changes. Comparing these controls with the existing curved layouts would help distinguish sensitivity to spacing and proportions from the additional difficulty introduced by bending the row-column structure. The Fragment Matching task includes nonuniform line-spacing patterns (Appendix A, Table 3), but does not substitute for a matched manipulation of spacing on the same underlying instances.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2QCD

We thank Reviewer 2QCD for recognizing that our work identifies a meaningful limitation in how existing benchmarks assess visual reasoning, and that Polaris-Bench offers a valuable diagnostic perspective.

 

Q1 & W: Mechanistic attribution of the Cartesian–Polar gap, the two-stage analysis, and the text-only baseline.

A: We respond to this question regarding attribution and the two-stage experiment in overall response. Besides, consistent with the reviewer's point, and as noted in Sec. 2 (where coordinate discretization "demonstrates robust deductive capability"): using explicit coordinates on orthogonal grids is a legitimate and valuable problem-solving abstraction rather than a model flaw; we frame the Cartesian Shortcut from a visual-reasoning evaluation perspective, as orthogonal grids allow text-based deduction to inflate estimates of general visual capability. We appreciate the reviewer's suggestion on our writing and will revise the corresponding wording in the paper to more clearly separate "the observed layout sensitivity" from "the mechanistic interpretation."

AuthorsOpenReview ↗

Response to Reviewer 2QCD [part 2]

 

Q2&W: Orthogonal Structure vs. Uniform Spacing and Cell Proportions

A: Following reviewer’s suggestion, we added the two straight-line orthogonal controls to the geometric deformation experiment in Section 5 / Appendix D (Table 21) on the same three tasks and six models. Specifically, 

  • (1) Rect uniformly stretches either the row or column axis by a random factor of 2×, 3×, or 4× into rectangular cells, 
  • (2) Non-unif randomly perturbs each row height and each column width within 0.4×~2.0× to break uniform spacing. Both conditions enforce a minimum cell-width floor so that legibility is unaffected. 

The table below reports the full comparison with Rect and Non-unif added to Table 21.

ModelCartesianRectNon-unifWaveArc-45°Arc-90°Arc-135°Arc-180°Polar
Gemini-3.1-Pro89.067.037.044.035.020.015.316.317.1
Gemini-3-Flash51.348.037.723.930.721.115.816.523.3
GPT-5.265.067.349.029.030.324.821.020.316.7
Claude-Sonnet-4.618.716.713.012.316.313.714.712.015.7
Gemma-4-31B27.730.326.717.722.315.919.216.318.0
Kimi-k2.526.726.723.316.019.018.314.714.015.3
  • Under Rect, most models (e.g., GPT-5.2) maintain accuracy comparable to the original Cartesian grid, while some models show a performance drop (e.g., Gemini-3.1-Pro, −22.0 pp). 
  • Under Non-unif, all models drop in accuracy relative to Cartesian (−1.0 to −52.0 pp), with particularly large drops on stronger frontier models (−13.6 to −52.0 pp). And performance under Non-unif drops close to the level of Wave. 
  • At the same time, we observe notable behavioral differences across models under these two perturbations. 

These results extend our geometric deformation analysis, showing that model performance is sensitive to both axis curvature and spacing regularity; this consistent with our broader finding that departures from standard orthogonal grid structure degrade performance across models.

Reviewer_vxxXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the strong performance of multimodal large language models on grid-based visual reasoning tasks depends on the orthogonal structure of Cartesian layouts. The authors refer to models’ tendency to discretize visual grids into textual representations such as rows, columns, and explicit coordinates, and subsequently rely on language-based deduction, as the “Cartesian Shortcut.” To evaluate model performance when this structure is disrupted, the paper introduces Polaris-Bench, which reformulates 53 visual reasoning tasks into paired Cartesian and Polar versions while attempting to preserve their rules, semantics, and answers. Experiments on 14 proprietary and open-source MLLMs show that several frontier models decrease from approximately 69%–83% accuracy on Cartesian layouts to 31%–39% on Polar layouts, whereas human accuracy decreases by only 5.7 percentage points. The paper further reports that the benefits of high-reasoning inference are substantially reduced on Polar layouts, that conversion hints and five-shot in-context examples do not recover the lost performance, and that similar degradation occurs under other non-orthogonal layouts and progressive geometric deformations. Based on these results, the authors conclude that current MLLM visual reasoning performance is strongly dependent on orthogonal grid structure and propose Polaris-Bench as a diagnostic benchmark for studying robustness across geometric representations.

Critical strengths

S1) The paper offers a novel diagnostic perspective. Although prior work has studied perceptual failures, spatial reasoning limitations, and shortcut learning in MLLMs(see weakness), this paper revisits visual reasoning from the distinct perspective of coordinate representation and layout geometry, which reveals that standard orthogonal grids may provide models with an implicit structural advantage.

S2) The main empirical phenomenon is large and consistent. Several frontier models decrease from approximately 69%–83% accuracy on Cartesian layouts to 31%–39% on Polar layouts. The trend persists across model families, task categories, fully aligned tasks, and boundary conditions.

S3) The paper has substantial empirical scope and workload and provides a broad set of supplementary analyses. Polaris-Bench contains 53 tasks and 5,300 Cartesian–Polar pairs. The evaluation covers 14 proprietary and open-source models, five reasoning categories, and multiple reasoning configurations.These include a human reference, reasoning-mode comparisons, CoT analysis, a two-stage perception–reasoning experiment, prompting interventions, non-Polar layouts, and progressive geometric deformations. The appendix documents task definitions, generation procedures, prompts, evaluation methods, and per-task results. It also discusses tasks that do not exhibit a Cartesian–Polar gap.

Critical weaknesses

W1) The paper overstates the conceptual novelty of the Cartesian Shortcut. Prior work has already shown that VLMs may drift away from visual evidence during multi-step reasoning and rely heavily on language priors, textual CoT, and the reasoning capabilities of their language backbones. Some studies have demonstrated this behavior through no-image ablations, controlled cross-modal comparisons, and attention analyses. Cartesian Shortcut seems to be a grid-specific instance of this broader phenomenon.

W2) The paper does not disentangle perceptual difficulty from reasoning difficulty under the Polar transformation. The transformation both makes visual elements and spatial relations harder to perceive and prevents straightforward conversion into textual coordinates. Since grid-size recognition is only a coarse proxy for perception, local reading errors remain uncontrolled. The experiments therefore cannot determine whether the degradation arises from impaired visual perception or from the inability to reason through textual coordinates.

W3) The paper lacks causal evidence for the effect of coordinate textualization. The CoT analysis only shows that models frequently use coordinate language; it does not establish that this improves Cartesian accuracy or causes Polar degradation.

Questions for authors

Q1) How do the authors distinguish the Cartesian Shortcut from previously documented language-dominated visual reasoning? Prior work has shown that VLMs may attend less to visual evidence during extended CoT and rely increasingly on language priors or textual reasoning. What additional mechanistic claim does the Cartesian Shortcut make? Is it a grid-specific instance of this broader phenomenon, or a separately identifiable mechanism?

Q2) How do the authors distinguish whether the performance degradation under the Polar transformation arises from perceptual difficulty or from the inability to reason through textual coordinates? The current experiment uses grid-size recognition as a proxy for perceptual correctness, but this does not ensure that cell contents, paths, and adjacency relations are correctly perceived. Could the authors separately evaluate fine-grained visual parsing and reasoning performance when a complete structured representation of ground truth is provided?

Q3) Can the authors provide causal evidence that explicit coordinate textualization improves Cartesian performance? The current CoT analysis only establishes that coordinate expressions frequently occur. Could the authors compare coordinate usage between successful and unsuccessful examples, or intervene by enforcing, prohibiting, or perturbing coordinate descriptions to test whether textualization causally affects accuracy?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vxxX

We thank Reviewer vxxX for recognizing our novel diagnostic perspective on layout geometry, the large and consistent empirical phenomenon, and the substantial scope of our evaluation and analyses.

 

Q1 & W1: Relationship to Prior Literature and Scope of the work

A: Shared motivation & Positioning of the Cartesian Shortcut: Our work shares the same core motivation as the prior studies noted by the reviewer: as MLLMs rapidly saturate visual benchmarks, do high scores reflect genuine visual reasoning, or heavy reliance on language priors and text-based deduction? In Sections 1 and 6, we cite and discuss these related directions: including benchmark leakage and image-free reasoning (Asadi et al., 2026), reduced visual grounding in reasoning modes (Xu et al., 2025), and perceptual or shortcut limitations (Xia et al., 2025); and approach this shared question from a complementary angle. Specifically, while prior studies along this line primarily examine cross-modal reliance (e.g., visual grounding versus language priors and textual reasoning), our work, as the reviewer noted in S1, revisits visual reasoning from "the distinct perspective of coordinate representation and layout geometry." Regarding mechanistic claims, consistent with our Overall Response, we do not claim that the Cartesian Shortcut is a separately identifiable internal mechanism; rather, while models' reliance on textual deduction connects to the broader phenomenon above, our framing focuses on a specific geometric property of orthogonal grid layouts.

Layout geometry as a diagnosis probe: Given the ubiquity of Cartesian grids across both real-world settings and visual reasoning benchmarks, their natural alignment with mathematical 2D matrices makes visual layouts straightforward to express as explicit coordinates for text-based deduction, consistent with what S1 describes as an "implicit structural advantage." Because this Cartesian advantage is tied to orthogonal layout structure, we probe layout dependence through semantics-preserving geometric transformations: as stated in our Conclusion (Sec. 7), Polar conversion is not the only such deformation (Sec. 5 also evaluates Wave, Arc, and Hexagonal layouts), and we select the Cartesian–Polar pair because this bijective mapping offers clean mathematical symmetry and remains intuitive to humans (Table 1), serving as a principled diagnostic probe. Across 53 paired tasks and 14 MLLMs, this probe reveals that both frontier models' high accuracy and their reasoning-mode gains are strongly coupled to orthogonal grid structure, and cannot be recovered by prompt-level conversion hints or 5-shot ICL examples (Sec. 4&5). 

In the revision, we will refine the phrasing in Sections 1 and 6 (e.g., replacing "specific mechanisms" in Sec. 6) to clarify how our layout-geometry perspective complements prior cross-modal grounding studies.

AuthorsOpenReview ↗

Response to Reviewer vxxX [part 2]

Q2&W2: Perceptual vs. Reasoning Difficulty, the Two-Stage Analysis, and Explicit Coordinate Controls

A: For the attribution and two-stage experiment aspects of this question, please see our Overall Response. Below we address the remaining parts:

Regarding visual perception on Polar layouts: 

(1) during task implementation, we calibrated our rendering pipeline during benchmark construction to ensure legibility (Section 3.2, Appendix B.3), which was also verified by human annotation (Appendix B.5). 

(2) Meanwhile, following the reviewer's suggestions (Q2 and Q3), to further explore how explicit coordinate information affects model performance, we conducted a paired experiment on five tasks (Template Matching, Grid Folding, Largest Number Path, Lattice Paths, and Path Counting) across six representative models. 

  • P: original Polar layout;
  • P+axis: labeling explicit ring and sector numbers along the Polar axes;
  • P+cell: labeling explicit (ring, sector) coordinates inside each Polar cell;
  • C: original Cartesian layout;
  • C+WD: stating wrong grid dimensions in the prompt;
  • C+WC: labeling shuffled wrong (row, col) coordinates in every cell;
  • C+SC: repeating the same random wrong coordinate in every cell as a control.

All conditions use the exact same paired instances (100 pairs per task) and a unified, non-occluding label style.

ModelCC+WDC+WCC+SCPP+axisP+cell
Gemini-3.1-Pro87.684.462.681.435.738.253.8
Gemini-3-Flash76.464.055.868.236.435.643.4
GPT-5.287.468.057.676.435.638.447.0
Claude-Sonnet-4.642.734.034.233.625.422.023.6
Gemma-4-31B67.863.252.465.229.233.835.2
Kimi-k2.566.143.051.656.429.129.831.4
Average71.359.452.463.531.933.039.1
  • On Polar layouts, P+axis remains close to P (33.0% vs. 31.9%), while P+cell raises the six-model average to 39.1% (+18.1 pp on Gemini-3.1-Pro and +11.4 pp on GPT-5.2, vs. −1.8 to +7.0 pp on the other four models), compared with 71.3% on the Cartesian baseline C. The gains on stronger models under P+cell are directionally consistent with our observations on Monotonic Path (Section 5) and the digit-labeled pilot version of Four Color (Appendix B.4) that in-cell numerical anchors can aid text-based tracking on Polar layouts, though unlike task-intrinsic cell values in those two tasks, generic coordinate labels here recover only part of the Cartesian–Polar gap.
  • Thus, providing ground-truth cell coordinates on Polar layouts partially aids stronger models, yet a substantial Cartesian-Polar gap (39.1% vs. 71.3% on average) persists across all models. The Cartesian perturbation columns (C+WD, C+WC, C+SC) are discussed in Q3 below (full results will be added to Section 5 and Appendix D).
AuthorsOpenReview ↗

Response to Reviewer vxxX [part 3]

Q3 & W3: Empirical Role of Coordinate Textualization and Coordinate Interventions

A: Besides our explanation in Overall Response (Scope, 2-stage experiment), for this question, following the reviewer's suggestion, we report the Figure 1 (left) CoT coordinate-usage rates separately for correct and incorrect predictions across all 53 tasks (table below): coordinate usage is higher on correct than incorrect Cartesian examples (substantially on Gemini-3.1-Pro and modestly on GPT-5.2), while remaining around 50%-64% on Polar layouts without a consistent direction across models.

ModelCartesian (Correct)Cartesian (Incorrect)Polar (Correct)Polar (Incorrect)
Gemini-3.1-Pro86.0%44.2%56.2%50.1%
GPT-5.289.0%81.8%53.3%64.0%

To further examine how Cartesian accuracy responds to coordinate interventions, we conducted two targeted studies (will add to Appendix D):

  • Coordinate perturbations (Columns C, C+WD, C+WC, and C+SC in the Q2 table above): In the paired five-task, six-model experiment reported in Q2, accuracy drops across all six models under all three Cartesian perturbations relative to C (71.3% on average, vs. 63.5% on C+SC, 59.4% on C+WD, and 52.4% on C+WC), with shuffled wrong coordinates (C+WC) yielding a larger drop than the repeated-coordinate control (C+SC) in five of the six models.
  • Prompt-based coordinate prohibition: Across 10 Cartesian tasks (1,000 instances), we evaluated Gemini-3.1-Pro and Gemini-3-Flash under an explicit prompt instruction prohibiting any row/column or coordinate indexing. Both models continued to invoke coordinates across their reasoning steps and final outputs in 86.4% and 96.3% of instances, respectively, and this prompt constraint lowered 10-task mean accuracy from 90.0% to 78.9% (−11.1 pp) on Gemini-3.1-Pro and from 81.9% to 71.2% (−10.7 pp) on Gemini-3-Flash.

Together, perturbing or restricting explicit coordinate cues degrades Cartesian accuracy across the tested tasks, while prompt-level prohibition alone fails to suppress models' strong tendency to invoke coordinates.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response to AC and reviewers

We sincerely thank the AC for coordinating the review process and all four reviewers for their thoughtful evaluations and constructive feedback. We are encouraged that the reviewers recognized our diagnostic perspective on how orthogonal-grid benchmarks may overestimate visual reasoning (vxxX, 2QCD, zKVM), our controlled paired design separating fully aligned from boundary-affected tasks (6yAK, 2QCD, zKVM), and the large, consistent empirical signal across 53 tasks, 14 MLLMs, and diverse geometric deformations (vxxX, 6yAK, zKVM). 

Below, we first provide a unified response to two shared questions across the reviews (mechanistic attribution and the two-stage experiment), followed by detailed answers and additional analyses in each individual response.

 


Scope and Mechanistic Attribution. Several reviewers (vxxX, 2QCD, 6yAK, zKVM) asked whether our experiments causally attribute the performance drop to coordinate textualization and disentangle perception, reasoning, etc. To clarify our scope, this paper establishes and studies two empirical facts:

  •  (a) The Cartesian Shortcut exists as a viable and practically valuable problem-solving pattern: on orthogonal layouts, models frequently discretize visual grids into explicit textual coordinates (>56% across 9 existing benchmarks and 80%-88% of CoTs on our Cartesian tasks) and achieve high accuracy through a caption-then-solve pathway alone (Section 2, Figure 1). We use "Cartesian Shortcut" to name this observed behavior, not as a causal claim for model performance or the Polar collapse.
  •  (b) Model performance is strongly coupled to orthogonal structure. We construct Polaris-Bench, reformulating 53 visual reasoning tasks into paired Cartesian and Polar layouts with preserved task semantics (10,600+ instances, Section 3), and show through a systematic empirical study across 14 MLLMs that when orthogonal grid structure is disrupted, frontier model accuracy collapses (69%-83% to 31%-39%) whereas human accuracy remains robust (94.5% to 88.8%, Table 1), with consistent drops across fully aligned tasks and non-Polar deformations (Sections 4-5).

Together, (a) and (b) support our central conclusion that benchmarks dominated by orthogonal grids may overestimate general visual reasoning. This scope was stated in the submission (Sec. 7: "We do not claim that Polar conversion is either necessary or sufficient for performance collapse … Rather, this work establishes the scope and severity of models' dependence on orthogonal structure"). 

Rather than claiming causal attribution to any single factor (where isolating contributions of visual perception, coordinate discretization, and familiarity requires dedicated experimental designs beyond our diagnostic scope), we leave disentangling these factors to dedicated follow-up studies that Polaris-Bench is designed to enable (Section 7: “separating the contributions of familiarity, geometry, and orthogonal structure ... remain important open directions”). Importantly, the core findings, conclusions and contributions of this work stand without establishing this causal relationship and do not depend on it. Following Reviewers 6yAK (W1) and 2QCD (W1)’s constructive framing suggestions, we will refine the wording in Section 2, Section 6, and Appendix C.2 to keep the demonstrated layout sensitivity clearly separated from mechanistic interpretations.

AuthorsOpenReview ↗

Overall Response to AC and reviewers [part 2]

 

Two-Stage Experiment. We provide the following response and further clarification on the two-stage experiment raised across the reviews (vxxX W2/Q2, 2QCD W1/Q1, 6yAK W1/Q1, zKVM W1):

  • (a) Purpose: The core purpose of this experiment (Section 2, Figure 1, Appendix C.2) is to support Point (a) above by showing that the model performs strongly across both stages on Cartesian layouts, with Polar included as a reference. This experiment does not attempt to make a causal attribution.
  • (b) Filter: This experiment was run on the full benchmark of 10,600 instances across all 53 tasks. At this scale, human verification of caption accuracy is impractical; image-grounded MLLM verification would inherit the very perceptual degradation under evaluation. Matching extracted grid dimensions against ground-truth metadata therefore provides a reasonable, scalable negative filter for Stage 1. Specifically, a correct grid size does not guarantee that perception is accurate, but an incorrect grid size indicates an obvious error in perception; grid size thus serves as a negative filter for Stage 1, with Stage 2 providing an indicative signal on the remaining instances.
  • (c) Following the reviewers' questions and suggestions (6yAK Q1, 2QCD W1, vxxX W2, zKVM W1), we re-ran the Stage 2 examples using the original images instead of captions (including on the 983 paired questions) to exclude caption inaccuracies and unpaired subset bias (Table below). The image-based results closely match the caption-based results across both subsets, showing that captions alone suffice on Cartesian while the ~40-point Polar drop holds without caption bottlenecks or pairing bias.
  • (d) Regarding an oracle pure-text baseline, as Reviewer 2QCD noted, an oracle transcription "would not by itself establish that textual coordinate use causally explains the image-based gap": Indeed, once an image is fully expressed as structured text, we cannot prevent the model from directly leveraging its text-based matrix/Cartesian reasoning capabilities to solve the problem regardless of original layout, making it difficult to cleanly attribute the image-based gap in this setting as well.
SubsetStage 2 InputCartesianPolar
Stage-1 FilteredCaption86.9%47.6%
Stage-1 FilteredOriginal Image86.6%46.3%
Paired Subset (983 pairs)Caption88.2%48.6%
Paired Subset (983 pairs)Original Image90.2%46.6%

 

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.