- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This article proposes Polaris-Bench, which divides 53 types of visual reasoning tasks into two versions, grid and polar coordinates, to test the model's adaptability to layout changes.
- Several strong models dropped significantly on the polar coordinate version, and increasing thinking and providing examples failed to effectively bridge the gap. The authors believe that existing models rely too much on regular grids and their textual coordinate representation.
Critical strengths
- The starting point of the question is interesting. If you change the layout of the same question, you will obviously lose points, which shows that the conventional visual reasoning assessment may indeed miss important issues.
- A paired experimental design is better. The author distinguishes tasks with complete logical alignment and topological changes, reducing the interference of "actually the topic has changed".
- The experiment covers a wide range of areas. Containing multiple strong models, inference patterns, and other geometric deformations, the observed phenomena are not isolated cases of a single model or torus layout.
Critical weaknesses
- Perception and reasoning are not sufficiently separated. Correctly identifying the grid size does not mean correctly reading the grid content and adjacency relationships. Therefore, the existing experiments are insufficient to illustrate how problematic reasoning itself remains after perceptual errors are ruled out.
- The influence of training distribution is not excluded. The model may simply have seen less polar tasks. In the absence of training or adaptation experiments, it is unclear whether this is a shortcoming that can be replenished through data, or a deeper capability limitation.
- The actual mission significance is unclear. The experiment mainly focuses on artificially generated grid questions. Whether it can reflect the problems in real charts, maps or other visual scenes still needs to be verified.
Questions for authors
- Can we provide the correct lattice content and adjacency relationships, and further measure how many inference errors are left in the model after excluding visual reading errors?
- How much could the performance gap be reduced by adding a small amount of polar training data, or adding explicit numbers to rings and sectors?
- Can a similar phenomenon be verified on real graphs or maps, showing that this finding is not limited to synthetic grid tasks?
5 comments
Response to Reviewer zKVM [part 2]
Q2 & W2: Training Distribution and Explicit Ring/Sector Numbers on Polar Layouts
A: Regarding training: We share the reviewer’s view that training-level interventions are a valuable direction to further explore, and to help disentangle training-distribution familiarity from deeper capability limitations, which aligns with paper Section 7 ("separating the contributions of familiarity, geometry, and orthogonal structure through training-level interventions ... remain important open directions"). Consistent with our Overall Response, this paper focuses on identifying the orthogonal-grid evaluation confound, constructing Polaris-Bench, and empirically characterizing layout sensitivity across off-the-shelf MLLMs, conclusions that hold without claiming a deeper capability limitation. Parameter-level training is therefore outside the scope of this diagnostic study and remains a promising future direction that Polaris-Bench is designed to facilitate as a testbed.
Regarding adaptation experiments: On the test-time adaptation side, we evaluated 5-shot in-context learning with solved Polar examples and explicit Cartesian-conversion hints across six models and six tasks (Section 5, Figure 7(f), and Appendix D.3 Table 20), finding that neither recovers Polar accuracy (changes within <4 pp for hints, and lower accuracy on five of six models for 5-shot ICL).
Regarding explicit ring/sector numbers on Polar: Following Reviewer zKVM’s suggestion (combined with coordinate-perturbation suggestions from other reviewers), we conducted a paired experiment on five tasks (Template Matching, Grid Folding, Largest Number Path, Lattice Paths, and Path Counting) across six representative models:
- P: original Polar layout;
- P+axis: labeling explicit ring and sector numbers along the Polar axes;
- P+cell: labeling explicit (ring, sector) coordinates inside each Polar cell;
- C: original Cartesian layout;
- C+WD: stating wrong grid dimensions in the prompt;
- C+WC: labeling shuffled wrong (row, col) coordinates in every cell;
- C+SC: repeating the same random wrong coordinate in every cell as a control.
All conditions use the exact same paired instances (100 pairs per task) and a unified, non-occluding label style. The table below reports the five-task mean accuracy (%):
| Model | C | C+WD | C+WC | C+SC | P | P+axis | P+cell |
|---|---|---|---|---|---|---|---|
| Gemini-3.1-Pro | 87.6 | 84.4 | 62.6 | 81.4 | 35.7 | 38.2 | 53.8 |
| Gemini-3-Flash | 76.4 | 64.0 | 55.8 | 68.2 | 36.4 | 35.6 | 43.4 |
| GPT-5.2 | 87.4 | 68.0 | 57.6 | 76.4 | 35.6 | 38.4 | 47.0 |
| Claude-Sonnet-4.6 | 42.7 | 34.0 | 34.2 | 33.6 | 25.4 | 22.0 | 23.6 |
| Gemma-4-31B | 67.8 | 63.2 | 52.4 | 65.2 | 29.2 | 33.8 | 35.2 |
| Kimi-k2.5 | 66.1 | 43.0 | 51.6 | 56.4 | 29.1 | 29.8 | 31.4 |
| Average | 71.3 | 59.4 | 52.4 | 63.5 | 31.9 | 33.0 | 39.1 |
- On Polar layouts, relative to the 39.4 pp six-model average C–P gap (C: 71.3% vs. P: 31.9%), P+axis remains close to P (33.0%, narrowing the gap by 1.1 pp), while P+cell raises the six-model average to 39.1% (narrowing the gap by 7.2 pp overall: +18.1 pp on Gemini-3.1-Pro and +11.4 pp on GPT-5.2, vs. −1.8 to +7.0 pp on the other four models). The gains on stronger models under P+cell are directionally consistent with our observations on Monotonic Path (Section 5) and the digit-labeled pilot version of Four Color (Appendix B.4) that in-cell numerical anchors can aid text-based tracking on Polar layouts, though unlike task-intrinsic cell values in those two tasks, generic coordinate labels here recover only part of the Cartesian-Polar gap.
- On Cartesian layouts, all six models drop in accuracy under all three perturbations relative to C (averaging 63.5% on C+SC, 59.4% on C+WD, and 52.4% on C+WC, down from 71.3%), and shuffled wrong coordinates (C+WC) cause a larger drop than the repeated-coordinate control (C+SC) in five of the six models.
- Overall, on these five tasks, perturbing explicit coordinates lowers Cartesian accuracy, whereas providing ground-truth coordinates on Polar layouts improves performance for some models (particularly stronger ones), though a substantial Cartesian-Polar gap remains across all models.
Response to Reviewer zKVM [part 3]
Q3 & W3: Practical Significance and Connection to Real-World Graphs, Maps, and Visual Scenes
A: We deliberately adopt procedural generation to construct a controlled paired testbed, which is essential for isolating coordinate-layout effects under aligned task rules, a level of control that cannot be achieved with naturally occurring images. Because such pair-wise data is difficult to collect from naturally occurring images, we use procedural synthesis to build this controlled comparison.
Connection to real-world scene: While Polaris-Bench is constructed in a synthetic environment, the layout sensitivity it reveals has direct implications for both benchmark evaluation and real-world applications.
- First, current visual reasoning benchmarks rely heavily on orthogonal grid layouts (Section 2: over 56% of examples across 9 benchmarks elicit explicit textual coordinate discretization), indicating that existing evaluations may overestimate model robustness across broader visual layouts.
- Second, as the reviewer noted, many real-world visual scenarios naturally lack orthogonal grid structure, such as analog clocks and dials, circular or radar charts, and transit or geographic maps. Recent empirical studies and benchmarks on real-world scenarios report observations consistent with our Cartesian–Polar findings, showing that frontier MLLMs also struggle on real-world tasks with non-orthogonal or complex spatial layouts, including analog clock reading (e.g., Lost in Time (Saxena et al, 2025), pointer meter and dial reading (e.g., DialBench, Wang et al., 2025), non-orthogonal chart understanding such as radar and rose charts (e.g., ChartX, Xia et al., 2024), and route planning or spatial reasoning on real transit and cartographic maps (e.g., FRIEDA, Pyo et al., 2026).
Accordingly, our controlled Cartesian–Polar testbed provides a diagnostic reference for evaluating layout robustness, and serves as a foundation for future work to connect controlled layout transformations with real-world visual scenarios and study the underlying mechanisms.
References:
- Saxena, Rohit, Aryo Pradipta Gema, and Pasquale Minervini. "Lost in time: Clock and calendar understanding challenges in multimodal LLMs." arXiv preprint arXiv:2502.05092 (2025).
- Wang, Futian, et al. "DialBench: Towards Accurate Reading Recognition of Pointer Meter using Large Foundation Models." arXiv preprint arXiv:2511.21982 (2025).
- Xia, Renqiu, et al. "Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning." IEEE Transactions on Image Processing (2025).
- Pyo, Jiyoon, et al. "FRIEDA: Benchmarking multi-step cartographic reasoning in vision-language models." International Conference on Learning Representations. Vol. 2026. 2026.
Response to Reviewer zKVM
We thank Reviewer zKVM for noting that conventional visual reasoning evaluation may miss important issues, and for recognizing our paired design and broad coverage across models and layouts.
Q1 & W1: Perception vs. Reasoning Separation, the Two-Stage Analysis, and Ground-Truth Textual Representation
A: Please see our Overall Response (Scope and Mechanistic Attribution and Two-Stage Experiment) for a unified response. Specifically, our paired/image-based Stage-2 control and the 983-pair analysis (Overall Response, Table) show that the ~40pp Cartesian–Polar gap holds when caption bottlenecks and pairing bias are excluded. Regarding the pure-text setting, as discussed in Overall Response (d), fully expressing an image as structured text would allow the model to bypass visual processing entirely and leverage text-based matrix reasoning regardless of the original layout. The explicit-coordinate experiments on Polar layouts (P+axis and P+cell, Q2 below) also partially address this question.