- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TangramPuzzle, a benchmark for evaluating compositional spatial reasoning under strict geometric constraints in multimodal large language models. It contains 668 configurations and 1,336 task instances. The two tasks require models to identify an overall silhouette from the geometry of its constituent pieces and to generate a complete tangram assembly from a target silhouette, respectively. TCE provides a structured geometric representation, while the verifier separately checks syntax, rigid geometry, overlap and connectivity, and silhouette matching. The evaluation accepts different valid internal assemblies. The experiments reveal a substantial gap between producing a visually similar silhouette and constructing a geometrically valid assembly.
Critical strengths
1.A sound evaluation design that accepts multiple valid solutions. Task 2 separates fidelity from validity, exposing piece deformations, overlaps, and disconnected configurations that silhouette similarity alone may miss. Evaluating geometric constraints and target coverage also avoids penalizing valid assemblies with different internal decompositions (Section 3.3).
2.Analyses beyond a single success rate. The paper includes ICL, CoT, decomposition, modality ablations, TCE-Light, human comparisons, and separate overlap and connectivity analyses. These experiments help characterize the benchmark’s difficulties (Tables 3–4 and Appendices I–K).
Critical weaknesses
-
The evaluation benefits over the closest prior work need clarification. Section 2.2, Table 1, and Appendix A describe differences in task format, but do not fully explain the additional capabilities these differences allow the benchmark to assess. For example, MARBLE’s M-Cube already involves rotating and flipping fixed pieces, complete assembly, and validity checking, yet is absent from the comparison. Please clarify which capabilities or failure modes become measurable through arbitrary two-dimensional silhouettes, explicit coordinate outputs, and different internal assemblies. The comparison currently emphasizes design features; their concrete evaluation benefits need clearer explanation. Evaluating a few recent representative multimodal models, including open-weight models, would also help assess the benchmark’s ability to distinguish current models.
-
The evidence against superficial pattern matching in Task 1 is insufficient. Appendix H.1 uses low model performance to support the view that Task 1 is difficult to solve through memorization or superficial pattern matching. However, low scores alone do not rule out simpler discriminative strategies. Task 1 provides the final piece coordinates, while Appendix D samples distractor silhouettes randomly. Some questions might therefore be distinguishable using aspect ratio or local features such as a pointed or flat top, without reconstructing the complete silhouette. For instance, aspect ratio follows directly from coordinate extrema, and local boundary features at the highest positions may eliminate candidates. These are untested alternatives, not demonstrated shortcuts. A simple feature-matching baseline or distractors matched on these features would strengthen the interpretation; alternatively, the capability claims could be narrowed.
Questions for authors
1.Relative to M-Cube’s rigid-piece assembly and validator, which additional capabilities or failure modes can TangramPuzzle independently reveal? Could the comparison distinguish globally symmetry-equivalent solutions from genuinely different internal decompositions and clarify these differences in Table 1?
2.Can simple features such as aspect ratio, local boundary features at extreme positions, or the area centroid distinguish some of the randomly sampled candidates? Does Task 1 retain similar performance when these features are matched across candidates?
3.Could two or three recent representative multimodal models, including open-weight models, be evaluated to assess whether Task 1 is approaching saturation and whether Task 2’s strict success rate and auxiliary metrics continue to distinguish stronger models?
2 comments
Task 1 results with feature-matched distractors.
| Model | Original Task 1 | Feature-Matched (With Ticks) | Feature-Matched (Without Ticks) |
|---|---|---|---|
| GPT-6.1-sol | 99.85% | 99.70% | 99.40% |
| Qwen3.8-27B | 92.50% | 90.87% | 70.96% |
The results reveal distinct effects of feature-matched distractors and coordinate cues. With coordinate ticks retained, Qwen3.8-27B's accuracy decreases from 92.50% to 90.87% under feature-matched distractors. Removing the ticks further reduces its accuracy to 70.96%, whereas GPT-6.1-sol maintains consistently high performance across all settings. These findings suggest that both distractor similarity and coordinate cues influence model performance, while demonstrating the value of a more challenging Task 1 variant for evaluating models across different capability levels.
We will release this more challenging Task 1 variant alongside the benchmark, and revise Appendix H.1 to avoid claiming that low accuracy alone excludes superficial pattern matching. Instead, we will explicitly acknowledge potential feature-based shortcuts and present the new results as a more controlled assessment of silhouette discrimination.
Q3: Could two or three recent representative multimodal models, including open-weight models, be evaluated to assess whether Task 1 is approaching saturation and whether Task 2’s strict success rate and auxiliary metrics continue to distinguish stronger models?
We thank the reviewer for this suggestion. In response, we conducted additional evaluations of two recent representative multimodal models, GPT-6.1-sol and the open-weight Qwen3.8-27B, on all 668 instances of each task under the Vanilla setting.
1. Task 1: Evaluating Models Across Capability Levels. Task 1 is designed as a foundational assessment of compositional spatial reasoning and is intended to evaluate models across different capability levels, rather than exclusively targeting the strongest models. On the original Task 1, GPT-6.1-sol and Qwen3.8-27B achieve accuracies of 99.85% and 92.50%, respectively. To provide a more challenging evaluation, we additionally introduced the feature-matched distractor variant described in Q2. For Qwen3.8-27B, accuracy decreases to 90.87% with feature-matched distractors and further to 70.96% when coordinate ticks are removed. This two-tier evaluation design enables TangramPuzzle to assess foundational spatial reasoning across diverse model capability levels while providing a more demanding evaluation for stronger models.
2. Task 2: Distinguishing Advanced Geometric Assembly Capabilities. We evaluated both models on all 668 Task 2 instances, obtaining the following results:
Evaluation results of recent MLLMs on Task 2.
| Model | TSE↓ | RGE↓ | PE↓ | VPR↑ | IoU↑ | Hausdorff↓ | Success↑ |
|---|---|---|---|---|---|---|---|
| GPT-6.1-sol | 1.80 | 1.65 | 55.24 | 42.37 | 98.39 | 0.0205 | 42.22 |
| Qwen3.8-27B | 1.20 | 84.28 | 92.37 | 4.49 | 68.84 | 0.5586 | 4.19 |
These results demonstrate TangramPuzzle's ability to differentiate recent MLLMs through both strict success criteria and fine-grained geometric diagnostics. GPT-6.1-sol achieves 42.22% Success, substantially outperforming Qwen3.8-27B (4.19%), while Qwen3.8-27B's high RGE (84.28%) and PE (92.37%) reveal substantial difficulties in preserving piece rigidity and satisfying physical constraints. Despite improvements in recent models, reliable geometric assembly remains challenging, underscoring the continued value of TangramPuzzle as an evaluation benchmark.
Overall, these additional experiments show that Task 1 provides a useful foundational assessment across model capability levels, while Task 2 continues to distinguish stronger models and expose their remaining geometric reasoning limitations. We will incorporate these new evaluations and the enhanced Task 1 variant into the revised manuscript.
We sincerely thank Reviewer oUtA for the detailed comments and helpful suggestions; we respond to each concern in turn below.
Q1: Relative to M-Cube’s rigid-piece assembly and validator, which additional capabilities or failure modes can TangramPuzzle independently reveal? Could the comparison distinguish globally symmetry-equivalent solutions from genuinely different internal decompositions and clarify these differences in Table 1?
We thank the reviewer for identifying MARBLE's M-Cube as an important related benchmark. We agree that M-Cube already evaluates rigid-piece rotation, flipping, complete assembly, and solution validity. We will include it in Section 2.2, Table 1, and Appendix A, and clarify the additional evaluation capabilities enabled by TangramPuzzle rather than emphasizing task-format differences alone.
1. Continuous Geometric Assembly and Fine-Grained Error Diagnosis. While M-Cube evaluates rigid-piece assembly and validity in a structured 3D cube setting, TangramPuzzle requires models to generate explicit 2D coordinates to assemble rigid pieces into diverse target silhouettes. This enables independent evaluation of piece-level geometric rigidity (RGE), physical feasibility (PE), and global silhouette fidelity (IoU and Hausdorff distance). These metrics reveal distinct failure modes: a model may preserve individual piece geometry but fail to assemble the target silhouette, or approximate the target while deforming the pieces. Thus, TangramPuzzle provides complementary, fine-grained diagnostics of geometric assembly failures, particularly by separating rigid-piece preservation, physical feasibility, and silhouette matching.
2. Symmetry-Equivalent Solutions vs. Different Internal Assemblies. TangramPuzzle evaluates geometric validity without requiring the predicted internal arrangement to match a single ground truth (GT). To investigate alternative solutions, we further categorized Gemini3-Pro's 144 successful predictions under the Vanilla setting by comparing the placement of each piece type with the GT, allowing permutations of identical pieces and checking global symmetry equivalence separately. We found that 32 (22.2%) match the GT arrangement, 17 (11.8%) are equivalent under global rotation or reflection, and 95 (66.0%) represent distinct internal arrangements beyond global symmetry equivalence. This demonstrates that successful assemblies frequently differ from the reference beyond simple symmetry transformations. We also confirmed that the verifier accepts independently solver-generated alternative valid solutions without requiring agreement with the GT internal arrangement.
Together, these properties make TangramPuzzle complementary to M-Cube: it assesses continuous geometric precision, distinguishes piece-level validity from global silhouette fidelity, and evaluates alternative valid internal assemblies. We will revise Table 1 and the related discussion to explicitly connect these design differences to their measurable capabilities and failure modes.
Q2: Can simple features such as aspect ratio, local boundary features at extreme positions, or the area centroid distinguish some of the randomly sampled candidates? Does Task 1 retain similar performance when these features are matched across candidates?
We thank the reviewer for identifying potential shortcuts in Task 1. We agree that low model accuracy alone cannot rule out simple feature-based discrimination, and that randomly sampled distractors may sometimes be eliminated using coarse geometric cues rather than complete silhouette reconstruction.
To address this concern, we conducted additional experiments with a more challenging, feature-matched version of Task 1. Instead of randomly sampling distractors, we constructed a geometric descriptor for each of the 668 existing silhouettes, incorporating aspect ratio, bounding-box occupancy, convexity, perimeter-to-area ratio, horizontal and vertical width profiles, and boundary widths near the top, bottom, left, and right extremes. For each question, we selected three distractors that minimize the total pairwise descriptor distance among the four candidates, while excluding candidates with IoU ≥ 0.97 relative to the correct silhouette. We further evaluated versions with and without coordinate ticks to examine the effect of visual coordinate cues.