acceptodds
Under review as a conference paper at ICLR 2027

JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

Abstract

Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce , a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (44 to 1616), we find that zero-shot VLMs largely lack geometric reasoning: only one of five frontier models (GPT-5.5) exceeds random baseline on 44 puzzles, while all others perform at chance level. While supervised fine-tuning achieves 97% on 44, demonstrating the task is learnable, all models collapse on larger grids: GPT-5.5 drops from 70% to near-random on 88, and even fine-tuned models fall below 5% on 1212. This “scaling cliff” suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. establishes scalable geometric reasoning as an open challenge for vision-language models.

Primary area:
datasets and benchmarks
Keywords:
Visual-Geometric Reasoning, Vision-Language Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tQvXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

To address ambiguity from repetitive textures in rectangular-tile VLM benchmarks and their limited assessment of complex spatial reasoning, this paper introduces JigShape, a visual-geometric reasoning benchmark. It uses complementary tab-and-blank constraints and includes about 95,000 puzzles on grids from 4×4 to 16×16. Zero-shot evaluation, supervised fine-tuning, and ablation studies show that some models perform well on small puzzles, reaching over 97% accuracy after fine-tuning, but performance drops sharply as puzzle size grows. This reveals limitations in complex geometric reasoning and constraint satisfaction.

Critical strengths
  1. JigShape presents a novel approach to evaluating VLM spatial reasoning. Complementary geometric constraints offer a new setting for assessing joint visual and geometric reasoning.
  2. JigShape is large and spans multiple difficulty levels. Multi-model evaluations and ablation studies also demonstrate substantial experimental effort.
Critical weaknesses
  1. The constraint solver appears to verify only that a valid solution exists, not that it is unique. Even in 16×16 puzzles, the limited Tab–Blank edge types may permit multiple geometrically compatible arrangements.
  2. The sharp performance decline with increasing puzzle size (“Scaling Cliff”) is not sufficiently analyzed. The paper does not distinguish geometric reasoning deficits from visual-resolution limits, increased input complexity, or output-format errors, leaving the underlying cause unclear.
Questions for authors
  1. How does the proposed method guarantee a unique solution rather than merely reduce the number of valid arrangements? If uniqueness cannot be rigorously established, could the relevant claims be appropriately qualified?
  2. Could more detailed visual error analyses be provided, including breakdowns of output-format errors, visual recognition errors, and geometric constraint violations across puzzle sizes, to clarify the main causes of the Scaling Cliff?
2 comments

Q1: The constraint solver appears to verify only that a valid solution exists, not that it is unique. Even in 16×16 puzzles, the limited Tab–Blank edge types may permit multiple geometrically compatible arrangements.

Thank you for pointing out this distinction. We agree that the limited tab–blank edge types can permit multiple geometrically compatible arrangements, including in 16×16 puzzles.

JigShape combines geometric compatibility with pixel-level visual continuity. Pieces sharing the same edge signature may still differ in texture, color, and object-boundary continuity with their neighbors, so an alternative arrangement satisfying the shape constraints need not form a visually coherent reconstruction. These visual cues provide additional constraints for disambiguation.

We will revise the corresponding statements to distinguish geometric feasibility from uniqueness.

Q2: The sharp performance decline with increasing puzzle size (“Scaling Cliff”) is not sufficiently analyzed. The paper does not distinguish geometric reasoning deficits from visual-resolution limits, increased input complexity, or output-format errors, leaving the underlying cause unclear.

Thank you for this observation. The “scaling cliff” describes an empirical performance pattern. Our existing results help distinguish output formatting and local geometric compatibility from reconstruction accuracy, although they do not fully separate visual perception from global reasoning.

First, the decline occurs even when outputs remain parseable. Fine-tuned Qwen produces parseable JSON at all evaluated grid sizes, yet its piece accuracy drops from 97.28% at 4×4 to 0.40% at 16×16. JSON formatting failures therefore cannot explain this model’s decline.

Second, the existing metrics reveal that global reconstruction accuracy decreases much more sharply than local shape compatibility. For fine-tuned Gemma, the results are:

Grid sizeShape compatibility (%)Piece accuracy (%)
4×498.9597.82
8×887.0633.58
12×1262.104.57

Thus, substantial local geometric compatibility can coexist with poor recovery of the full arrangement. These metrics characterize different aspects of the output, rather than assigning each error to a single underlying cause. The shape/no-shape ablation additionally shows sensitivity to boundary cues, but does not independently isolate visual-resolution effects.

Increasing grid size increases the number of fragments and required placements; model-side image processing may also change the effective visual detail available per piece. Separating these effects requires controlling the effective input resolution independently of grid size. A reliable visual-recognition versus geometric-reasoning error breakdown would additionally require intermediate predictions or targeted diagnostic tasks, since an incorrect final placement alone does not identify which stage failed.

We will incorporate this cross-grid analysis and clarify that the observed scaling cliff characterizes performance under the evaluated configurations. The current evidence identifies a gap between local compatibility and global reconstruction, but does not attribute that gap exclusively to geometric reasoning deficits.

Thank you for the clarification. Your response has largely addressed my concerns. I believe the work merits a score of 3 and will maintain my original rating.

Reviewer_eGqEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JigShape, a benchmark for evaluating visual-geometric reasoning in vision-language models through jigsaw puzzles with interlocking geometric constraints. The benchmark contains over 95K instances across four grid sizes (4×4 to 16×16), with shape and no-shape variants to examine the role of geometric information. The authors evaluate five frontier VLMs in zero-shot settings and two supervised fine-tuned models using piece accuracy, exact match, adjacency accuracy, and shape compatibility. Results show that most zero-shot models perform near random, while supervised fine-tuning achieves high accuracy on smaller puzzles. However, performance degrades substantially as grid complexity increases, and removing geometric constraints significantly reduces the performance of fine-tuned models.

Critical strengths

A useful and scalable benchmark for visual-geometric reasoning. The paper introduces an interesting benchmark design that addresses the ambiguity of traditional rectangular jigsaw puzzles through interlocking geometric constraints. The benchmark covers multiple grid sizes and includes controlled shape-based ablations, enabling systematic evaluation of visual-geometric reasoning across different levels of complexity. Its large scale and programmatic construction make it a potentially useful resource for studying the limitations of VLMs in spatial reasoning.

Critical weaknesses

1. Limited novelty and scientific insights relative to prior work. Jigsaw-based visual reasoning has already been investigated in prior studies such as Jigsaw-R1 and Visual Jigsaw, which also explore RL post-training and demonstrate generalization to downstream visual reasoning tasks. While JigShape introduces interlocking geometric constraints and larger puzzle configurations, it is unclear whether these changes provide substantially new insights beyond existing jigsaw-based evaluations. The paper primarily establishes that current VLMs struggle as puzzle complexity increases, which is not particularly surprising given prior findings.

2. Limited experimental validation of the benchmark's broader utility. The experiments primarily focus on zero-shot evaluation and supervised fine-tuning of two models, without investigating more competitive training strategies such as RL with verifiable rewards or curriculum learning, which have already been explored in related work. Such experiments would be particularly relevant for understanding whether the observed performance degradation can be overcome through improved training. Furthermore, evaluating whether training on JigShape transfers to other visual or spatial reasoning benchmarks would help establish the broader utility of the proposed benchmark beyond measuring puzzle-solving accuracy.

Questions for authors
  1. Uniqueness verification (clarification): Appendix E.4 claims that the constraint-based solver verifies unique solutions. However, multiple pieces can share the same edge signature. How does the solver establish uniqueness rather than simply recovering one valid arrangement? Does it also consider visual content?

  2. Scaling limitations (clarification): Since invalid JSON responses receive zero accuracy, what proportion of model outputs are invalid or incomplete at each grid size? Could this partially explain the reported performance degradation on larger puzzles?

  3. Training strategy (clarification): Given that the experiments primarily use SFT, how confidently can the observed scaling limitations be attributed to VLM reasoning capabilities rather than the training approach? How might the findings differ under RL or curriculum-based training methods explored in prior work?

4 comments

Q1: Limited novelty and scientific insights relative to prior work. Jigsaw-based visual reasoning has already been investigated in prior studies such as Jigsaw-R1 and Visual Jigsaw, which also explore RL post-training and demonstrate generalization to downstream visual reasoning tasks. While JigShape introduces interlocking geometric constraints and larger puzzle configurations, it is unclear whether these changes provide substantially new insights beyond existing jigsaw-based evaluations. The paper primarily establishes that current VLMs struggle as puzzle complexity increases, which is not particularly surprising given prior findings.

Thank you for highlighting the connection to Jigsaw-R1 and Visual Jigsaw. We agree that jigsaw-based visual learning and its downstream benefits are established by these works. A central contribution of JigShape is extending evaluation to dense, interlocking puzzles at 4×4, 8×8, 12×12, and 16×16 resolutions, with up to 256 pieces. This enables systematic evaluation of visual-geometric reconstruction over a substantially larger range of puzzle sizes.

Simply increasing the grid density of rectangular puzzles does not address the ambiguity introduced by visually similar fragments. As images are divided into smaller pieces, visual content can become less distinctive, particularly in repeated-texture regions. JigShape supplements these visual cues with complementary tab-and-blank boundaries, providing explicit local compatibility constraints throughout the grid. Our contribution is therefore a scalable benchmark combining dense grids with geometric constraints, together with metrics that distinguish local compatibility from successful global reconstruction.

The resulting insight goes beyond a general decline with increasing difficulty. For example, on 8×8 puzzles, fine-tuned Gemma3-12B achieves 87.06% shape compatibility but only 33.58% piece accuracy and 0.40% exact match. Thus, high local geometric compatibility can coexist with poor global reconstruction. The shape/no-shape ablation further shows that strong performance on interlocking puzzles is highly sensitive to removing boundary cues. Together, these findings characterize the gap between exploiting geometric cues and recovering the full arrangement, which is the central diagnostic contribution of JigShape.

Q2: Limited experimental validation of the benchmark's broader utility. The experiments primarily focus on zero-shot evaluation and supervised fine-tuning of two models, without investigating more competitive training strategies such as RL with verifiable rewards or curriculum learning, which have already been explored in related work. Such experiments would be particularly relevant for understanding whether the observed performance degradation can be overcome through improved training.

Thank you for this clarification. Our experiments establish that increasing puzzle size presents a substantial challenge for the evaluated zero-shot and SFT models. They do not isolate the contribution of the training strategy from that of the models’ visual-geometric reasoning capabilities.

The SFT results nevertheless reveal a specific gap between local compatibility and global reconstruction. On 8×8 puzzles, fine-tuned Gemma achieves 87.06% shape compatibility but only 33.58% piece accuracy and 0.40% exact match. Learning to satisfy many local geometric constraints therefore does not by itself ensure recovery of the complete arrangement.

RL with verifiable rewards could potentially improve global consistency by rewarding correct placements or complete solutions. Curriculum learning could help models progressively learn to coordinate larger numbers of pieces. These approaches may improve performance and shift where degradation occurs, but their effectiveness on dense interlocking puzzles remains an empirical question. JigShape provides ground-truth arrangements, explicit compatibility constraints, and multiple grid sizes for evaluating these possibilities. Our experiments establish reference results for this evaluation; they do not establish the best performance attainable with alternative training strategies.

Q3: Furthermore, evaluating whether training on JigShape transfers to other visual or spatial reasoning benchmarks would help establish the broader utility of the proposed benchmark beyond measuring puzzle-solving accuracy.

Thank you for highlighting downstream transfer as an important aspect of benchmark utility. Our present experiments evaluate reconstruction performance within JigShape; downstream transfer has not been evaluated. The benchmark’s demonstrated utility is its ability to distinguish local geometric compatibility from global reconstruction across increasing puzzle sizes and controlled shape/no-shape conditions.

JigShape also provides programmatically generated training instances and verifiable reconstruction targets, making it a resource for investigating whether learning joint visual-geometric reconstruction benefits other tasks. Whether those benefits materialize requires downstream evaluation. We will distinguish this potential training benefit from the diagnostic utility established by our experiments, rather than infer transfer from puzzle-solving accuracy alone.

Q4: Uniqueness verification (clarification): Appendix E.4 claims that the constraint-based solver verifies unique solutions. However, multiple pieces can share the same edge signature. How does the solver establish uniqueness rather than simply recovering one valid arrangement? Does it also consider visual content?

Thank you for pointing out this distinction. The constraint-based solver uses only discrete tab/blank/flat edge signatures and does not consider visual content. It returns the first geometrically valid arrangement rather than checking whether alternative arrangements exist. We therefore agree that recovering a valid arrangement does not establish uniqueness.

JigShape combines two complementary sources of evidence: geometric compatibility and pixel-level visual continuity. Although pieces with identical edge signatures are interchangeable under shape constraints alone, they may differ in texture, color, and object-boundary continuity with neighboring pieces. Thus, a geometrically valid alternative need not constitute a visually coherent reconstruction. The intended task requires jointly satisfying these geometric and visual requirements, with shape constraints providing additional evidence for disambiguating dense puzzles.

Q5: Scaling limitations (clarification): Since invalid JSON responses receive zero accuracy, what proportion of model outputs are invalid or incomplete at each grid size? Could this partially explain the reported performance degradation on larger puzzles?

Thank you for raising this concern. We audited the saved outputs. GPT-5.5, GPT-5.4-mini, both Grok variants, and fine-tuned Qwen3-VL-8B produced parseable JSON at every evaluated grid size. Llama-4-Maverick’s parsing failure rates were 0.4%, 0.8%, and 0% at 4×4, 8×8, and 12×12, respectively. These results show that substantial performance degradation occurs even when outputs remain parseable. For example, Qwen’s piece accuracy decreases from 97.28% at 4×4 to 0.40% at 16×16 despite parseable JSON throughout.

We also clarify that our implementation attempts to recover piece-coordinate mappings from incomplete or non-JSON responses and awards partial credit for correct placements; JSON parsing failure does not automatically result in zero piece accuracy. We will clarify this evaluation procedure and provide the complete per-model, per-grid parsing statistics in the revised manuscript. These statistics concern JSON parseability; they do not imply that every parsed mapping is complete or correct.

Q6: Training strategy (clarification): Given that the experiments primarily use SFT, how confidently can the observed scaling limitations be attributed to VLM reasoning capabilities rather than the training approach? How might the findings differ under RL or curriculum-based training methods explored in prior work?

Our experiments establish that increasing puzzle size presents a substantial challenge for the evaluated zero-shot and SFT models. They do not isolate the contribution of the training strategy from that of the models’ visual-geometric reasoning capabilities.

RL with verifiable rewards could potentially improve global consistency by rewarding correct placements or complete solutions. Curriculum learning could help models progressively learn to coordinate larger numbers of pieces. These approaches may improve performance and shift where degradation occurs, but their effectiveness on dense interlocking puzzles remains an empirical question. JigShape provides ground-truth arrangements, explicit compatibility constraints, and multiple grid sizes for evaluating these possibilities. Our experiments establish reference results for this evaluation; they do not establish the best performance attainable with alternative training strategies.

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer eGqE,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #13815

Thanks for the rebuttal. I understand that the benchmark exposes limitations of current models for solving this task, but I don't think it necessarily provides new insight into VLM failure modes (compared to similar prior work), and learning to solve it may not lead to improvements in other, more important areas of VLM application. As such, I am keeping my score.

Reviewer_HBaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual-geometric reasoning in VLMs using jigsaw puzzles. The authors argue that existing jigsaw puzzles contain ambiguities arising from rectangular cuts and have low grid density (e.g., typically smaller than 5×5). They therefore propose JigShape, which features interlocking jigsaw pieces with tab-and-blank geometric constraints and puzzles with grid sizes ranging from 4×4 to 16×16. Both zero-shot evaluations and fine-tuning experiments are conducted to study models’ reasoning abilities on JigShape.

Critical strengths
  1. The authors identify limitations of typical jigsaw puzzles and address them by introducing tab-and-blank geometric constraints and increasing grid density.
  2. The paper evaluates VLMs in both zero-shot and supervised fine-tuning settings. Moreover, four metrics—PA, EM, AA, and SC—are proposed to quantify model performance.
Critical weaknesses
  1. The claim that “zero-shot VLMs largely lack geometric reasoning” is overly broad, as the authors only test models on jigsaw puzzles, which represent a narrow case of visual-geometric reasoning. Since the paper aims to evaluate visual-geometric reasoning, it would be helpful to include additional tasks, such as maze-solving.
  2. The evaluated models are all from 2025 or earlier and are therefore somewhat outdated. Could the authors evaluate more recent VLMs?
  3. The zero-shot evaluation needs further clarification. The jigsaw puzzles use images from public sources such as DIV2K and Unsplash, which may have been included in the evaluated VLMs’ training data. Although the models may not have been trained on these specific jigsaw puzzles, it is unclear how potential exposure to the source images during training might influence the findings.
Questions for authors

see "Weaknesses".

3 comments

Q1: The claim that “zero-shot VLMs largely lack geometric reasoning” is overly broad, as the authors only test models on jigsaw puzzles, which represent a narrow case of visual-geometric reasoning. Since the paper aims to evaluate visual-geometric reasoning, it would be helpful to include additional tasks, such as maze-solving.

Thank you for pointing out this overgeneralization. We agree that performance on jigsaw puzzles alone does not establish a general lack of geometric reasoning in VLMs. Our intended scope is visual-geometric reconstruction: arranging image fragments by jointly using visual continuity, boundary cues, and local compatibility constraints.

We will revise the abstract and corresponding statements throughout the paper to limit our conclusions to this setting.

Maze-solving would provide a complementary assessment of spatial reasoning, particularly navigation and path planning. We agree that evaluating across such tasks would be valuable for assessing the breadth of these limitations. Our present benchmark focuses on reconstruction, and we will explicitly acknowledge that its findings do not establish how the evaluated models perform on other geometric reasoning tasks.

Q2: The evaluated models are all from 2025 or earlier and are therefore somewhat outdated. Could the authors evaluate more recent VLMs?

Thank you for raising this concern. We would like to clarify that our zero-shot evaluation already includes models released in 2026, including GPT-5.5 (April 23, 2026), GPT-5.4 mini (March 17, 2026), and Claude Opus 4.8 (May 28, 2026).

Our two supervised fine-tuning baselines, Qwen3-VL-8B and Gemma 3 12B, were released on October 15, 2025, and March 12, 2025, respectively. These open-weight models serve as supervised baselines alongside the zero-shot evaluation of the 2026 models.

Q3: The zero-shot evaluation needs further clarification. The jigsaw puzzles use images from public sources such as DIV2K and Unsplash, which may have been included in the evaluated VLMs’ training data. Although the models may not have been trained on these specific jigsaw puzzles, it is unclear how potential exposure to the source images during training might influence the findings.

Thank you for raising this concern. In our experiments, “zero-shot” refers to evaluating models without task-specific fine-tuning or in-context solved examples for JigShape; it does not imply that the source images were absent from their pretraining data.

We cannot rule out prior exposure to DIV2K or Unsplash images, since the pretraining corpora of the evaluated proprietary models are not fully disclosed. JigShape constructs puzzle instances using randomly assigned tab-and-blank boundaries and shuffled piece IDs, so familiarity with a source image does not directly provide the required piece-ID-to-position mapping. Nevertheless, recognizing or recalling the original image could help reconstruct its global layout and thereby influence performance.

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer HBaU,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #13815

I thank the authors for their rebuttal and maintain my original assessment.

Comments on the paper

1 comment

We thank the reviewers for their constructive comments. We have addressed each point in our rebuttal and would be grateful if you could take a look when you have a chance. We are happy to clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.