- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
To address ambiguity from repetitive textures in rectangular-tile VLM benchmarks and their limited assessment of complex spatial reasoning, this paper introduces JigShape, a visual-geometric reasoning benchmark. It uses complementary tab-and-blank constraints and includes about 95,000 puzzles on grids from 4×4 to 16×16. Zero-shot evaluation, supervised fine-tuning, and ablation studies show that some models perform well on small puzzles, reaching over 97% accuracy after fine-tuning, but performance drops sharply as puzzle size grows. This reveals limitations in complex geometric reasoning and constraint satisfaction.
Critical strengths
- JigShape presents a novel approach to evaluating VLM spatial reasoning. Complementary geometric constraints offer a new setting for assessing joint visual and geometric reasoning.
- JigShape is large and spans multiple difficulty levels. Multi-model evaluations and ablation studies also demonstrate substantial experimental effort.
Critical weaknesses
- The constraint solver appears to verify only that a valid solution exists, not that it is unique. Even in 16×16 puzzles, the limited Tab–Blank edge types may permit multiple geometrically compatible arrangements.
- The sharp performance decline with increasing puzzle size (“Scaling Cliff”) is not sufficiently analyzed. The paper does not distinguish geometric reasoning deficits from visual-resolution limits, increased input complexity, or output-format errors, leaving the underlying cause unclear.
Questions for authors
- How does the proposed method guarantee a unique solution rather than merely reduce the number of valid arrangements? If uniqueness cannot be rigorously established, could the relevant claims be appropriately qualified?
- Could more detailed visual error analyses be provided, including breakdowns of output-format errors, visual recognition errors, and geometric constraint violations across puzzle sizes, to clarify the main causes of the Scaling Cliff?
2 comments
Thank you for the clarification. Your response has largely addressed my concerns. I believe the work merits a score of 3 and will maintain my original rating.
Rebuttal
Thank you for pointing out this distinction. We agree that the limited tab–blank edge types can permit multiple geometrically compatible arrangements, including in 16×16 puzzles.
JigShape combines geometric compatibility with pixel-level visual continuity. Pieces sharing the same edge signature may still differ in texture, color, and object-boundary continuity with their neighbors, so an alternative arrangement satisfying the shape constraints need not form a visually coherent reconstruction. These visual cues provide additional constraints for disambiguation.
We will revise the corresponding statements to distinguish geometric feasibility from uniqueness.
Thank you for this observation. The “scaling cliff” describes an empirical performance pattern. Our existing results help distinguish output formatting and local geometric compatibility from reconstruction accuracy, although they do not fully separate visual perception from global reasoning.
First, the decline occurs even when outputs remain parseable. Fine-tuned Qwen produces parseable JSON at all evaluated grid sizes, yet its piece accuracy drops from 97.28% at 4×4 to 0.40% at 16×16. JSON formatting failures therefore cannot explain this model’s decline.
Second, the existing metrics reveal that global reconstruction accuracy decreases much more sharply than local shape compatibility. For fine-tuned Gemma, the results are:
Thus, substantial local geometric compatibility can coexist with poor recovery of the full arrangement. These metrics characterize different aspects of the output, rather than assigning each error to a single underlying cause. The shape/no-shape ablation additionally shows sensitivity to boundary cues, but does not independently isolate visual-resolution effects.
Increasing grid size increases the number of fragments and required placements; model-side image processing may also change the effective visual detail available per piece. Separating these effects requires controlling the effective input resolution independently of grid size. A reliable visual-recognition versus geometric-reasoning error breakdown would additionally require intermediate predictions or targeted diagnostic tasks, since an incorrect final placement alone does not identify which stage failed.
We will incorporate this cross-grid analysis and clarify that the observed scaling cliff characterizes performance under the evaluated configurations. The current evidence identifies a gap between local compatibility and global reconstruction, but does not attribute that gap exclusively to geometric reasoning deficits.