acceptodds
Under review as a conference paper at ICLR 2027

TangramPuzzle: Benchmarking Geometric Constraint-Based Compositional Spatial Reasoning in MLLMs

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding, yet precise compositional spatial reasoning under geometric constraints remains underexplored. Existing benchmarks mainly assess coarse spatial relations and rarely support rigorous geometric verification or multiple valid solutions in constructive tasks. To address these limitations, we introduce TangramPuzzle, a benchmark comprising 668 validated configurations and 1,336 instances for evaluating compositional spatial reasoning under geometric constraints. We formulate the Tangram Construction Expression (TCE) to encode tangram configurations with machine-verifiable geometric specifications, together with a multi-solution-aware verifier that separately evaluates geometric validity and silhouette fidelity. TangramPuzzle covers two complementary evaluation tasks: part-to-whole Outline Prediction and whole-to-part End-to-End Assembly Generation. We conduct extensive evaluation experiments on advanced open-source and proprietary models, revealing an interesting insight: MLLMs tend to prioritize matching the target silhouette while neglecting geometric constraints, leading to distortions or deformations of the pieces.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Spatial Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oUtAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TangramPuzzle, a benchmark for evaluating compositional spatial reasoning under strict geometric constraints in multimodal large language models. It contains 668 configurations and 1,336 task instances. The two tasks require models to identify an overall silhouette from the geometry of its constituent pieces and to generate a complete tangram assembly from a target silhouette, respectively. TCE provides a structured geometric representation, while the verifier separately checks syntax, rigid geometry, overlap and connectivity, and silhouette matching. The evaluation accepts different valid internal assemblies. The experiments reveal a substantial gap between producing a visually similar silhouette and constructing a geometrically valid assembly.

Critical strengths

1.A sound evaluation design that accepts multiple valid solutions. Task 2 separates fidelity from validity, exposing piece deformations, overlaps, and disconnected configurations that silhouette similarity alone may miss. Evaluating geometric constraints and target coverage also avoids penalizing valid assemblies with different internal decompositions (Section 3.3).

2.Analyses beyond a single success rate. The paper includes ICL, CoT, decomposition, modality ablations, TCE-Light, human comparisons, and separate overlap and connectivity analyses. These experiments help characterize the benchmark’s difficulties (Tables 3–4 and Appendices I–K).

Critical weaknesses
  1. The evaluation benefits over the closest prior work need clarification. Section 2.2, Table 1, and Appendix A describe differences in task format, but do not fully explain the additional capabilities these differences allow the benchmark to assess. For example, MARBLE’s M-Cube already involves rotating and flipping fixed pieces, complete assembly, and validity checking, yet is absent from the comparison. Please clarify which capabilities or failure modes become measurable through arbitrary two-dimensional silhouettes, explicit coordinate outputs, and different internal assemblies. The comparison currently emphasizes design features; their concrete evaluation benefits need clearer explanation. Evaluating a few recent representative multimodal models, including open-weight models, would also help assess the benchmark’s ability to distinguish current models.

  2. The evidence against superficial pattern matching in Task 1 is insufficient. Appendix H.1 uses low model performance to support the view that Task 1 is difficult to solve through memorization or superficial pattern matching. However, low scores alone do not rule out simpler discriminative strategies. Task 1 provides the final piece coordinates, while Appendix D samples distractor silhouettes randomly. Some questions might therefore be distinguishable using aspect ratio or local features such as a pointed or flat top, without reconstructing the complete silhouette. For instance, aspect ratio follows directly from coordinate extrema, and local boundary features at the highest positions may eliminate candidates. These are untested alternatives, not demonstrated shortcuts. A simple feature-matching baseline or distractors matched on these features would strengthen the interpretation; alternatively, the capability claims could be narrowed.

Questions for authors

1.Relative to M-Cube’s rigid-piece assembly and validator, which additional capabilities or failure modes can TangramPuzzle independently reveal? Could the comparison distinguish globally symmetry-equivalent solutions from genuinely different internal decompositions and clarify these differences in Table 1?

2.Can simple features such as aspect ratio, local boundary features at extreme positions, or the area centroid distinguish some of the randomly sampled candidates? Does Task 1 retain similar performance when these features are matched across candidates?

3.Could two or three recent representative multimodal models, including open-weight models, be evaluated to assess whether Task 1 is approaching saturation and whether Task 2’s strict success rate and auxiliary metrics continue to distinguish stronger models?

2 comments

We sincerely thank Reviewer oUtA for the detailed comments and helpful suggestions; we respond to each concern in turn below.

Q1: Relative to M-Cube’s rigid-piece assembly and validator, which additional capabilities or failure modes can TangramPuzzle independently reveal? Could the comparison distinguish globally symmetry-equivalent solutions from genuinely different internal decompositions and clarify these differences in Table 1?

We thank the reviewer for identifying MARBLE's M-Cube as an important related benchmark. We agree that M-Cube already evaluates rigid-piece rotation, flipping, complete assembly, and solution validity. We will include it in Section 2.2, Table 1, and Appendix A, and clarify the additional evaluation capabilities enabled by TangramPuzzle rather than emphasizing task-format differences alone.

1. Continuous Geometric Assembly and Fine-Grained Error Diagnosis. While M-Cube evaluates rigid-piece assembly and validity in a structured 3D cube setting, TangramPuzzle requires models to generate explicit 2D coordinates to assemble rigid pieces into diverse target silhouettes. This enables independent evaluation of piece-level geometric rigidity (RGE), physical feasibility (PE), and global silhouette fidelity (IoU and Hausdorff distance). These metrics reveal distinct failure modes: a model may preserve individual piece geometry but fail to assemble the target silhouette, or approximate the target while deforming the pieces. Thus, TangramPuzzle provides complementary, fine-grained diagnostics of geometric assembly failures, particularly by separating rigid-piece preservation, physical feasibility, and silhouette matching.

2. Symmetry-Equivalent Solutions vs. Different Internal Assemblies. TangramPuzzle evaluates geometric validity without requiring the predicted internal arrangement to match a single ground truth (GT). To investigate alternative solutions, we further categorized Gemini3-Pro's 144 successful predictions under the Vanilla setting by comparing the placement of each piece type with the GT, allowing permutations of identical pieces and checking global symmetry equivalence separately. We found that 32 (22.2%) match the GT arrangement, 17 (11.8%) are equivalent under global rotation or reflection, and 95 (66.0%) represent distinct internal arrangements beyond global symmetry equivalence. This demonstrates that successful assemblies frequently differ from the reference beyond simple symmetry transformations. We also confirmed that the verifier accepts independently solver-generated alternative valid solutions without requiring agreement with the GT internal arrangement.

Together, these properties make TangramPuzzle complementary to M-Cube: it assesses continuous geometric precision, distinguishes piece-level validity from global silhouette fidelity, and evaluates alternative valid internal assemblies. We will revise Table 1 and the related discussion to explicitly connect these design differences to their measurable capabilities and failure modes.

Q2: Can simple features such as aspect ratio, local boundary features at extreme positions, or the area centroid distinguish some of the randomly sampled candidates? Does Task 1 retain similar performance when these features are matched across candidates?

We thank the reviewer for identifying potential shortcuts in Task 1. We agree that low model accuracy alone cannot rule out simple feature-based discrimination, and that randomly sampled distractors may sometimes be eliminated using coarse geometric cues rather than complete silhouette reconstruction.

To address this concern, we conducted additional experiments with a more challenging, feature-matched version of Task 1. Instead of randomly sampling distractors, we constructed a geometric descriptor for each of the 668 existing silhouettes, incorporating aspect ratio, bounding-box occupancy, convexity, perimeter-to-area ratio, horizontal and vertical width profiles, and boundary widths near the top, bottom, left, and right extremes. For each question, we selected three distractors that minimize the total pairwise descriptor distance among the four candidates, while excluding candidates with IoU ≥ 0.97 relative to the correct silhouette. We further evaluated versions with and without coordinate ticks to examine the effect of visual coordinate cues.

Task 1 results with feature-matched distractors.

ModelOriginal Task 1Feature-Matched (With Ticks)Feature-Matched (Without Ticks)
GPT-6.1-sol99.85%99.70%99.40%
Qwen3.8-27B92.50%90.87%70.96%

The results reveal distinct effects of feature-matched distractors and coordinate cues. With coordinate ticks retained, Qwen3.8-27B's accuracy decreases from 92.50% to 90.87% under feature-matched distractors. Removing the ticks further reduces its accuracy to 70.96%, whereas GPT-6.1-sol maintains consistently high performance across all settings. These findings suggest that both distractor similarity and coordinate cues influence model performance, while demonstrating the value of a more challenging Task 1 variant for evaluating models across different capability levels.

We will release this more challenging Task 1 variant alongside the benchmark, and revise Appendix H.1 to avoid claiming that low accuracy alone excludes superficial pattern matching. Instead, we will explicitly acknowledge potential feature-based shortcuts and present the new results as a more controlled assessment of silhouette discrimination.

Q3: Could two or three recent representative multimodal models, including open-weight models, be evaluated to assess whether Task 1 is approaching saturation and whether Task 2’s strict success rate and auxiliary metrics continue to distinguish stronger models?

We thank the reviewer for this suggestion. In response, we conducted additional evaluations of two recent representative multimodal models, GPT-6.1-sol and the open-weight Qwen3.8-27B, on all 668 instances of each task under the Vanilla setting.

1. Task 1: Evaluating Models Across Capability Levels. Task 1 is designed as a foundational assessment of compositional spatial reasoning and is intended to evaluate models across different capability levels, rather than exclusively targeting the strongest models. On the original Task 1, GPT-6.1-sol and Qwen3.8-27B achieve accuracies of 99.85% and 92.50%, respectively. To provide a more challenging evaluation, we additionally introduced the feature-matched distractor variant described in Q2. For Qwen3.8-27B, accuracy decreases to 90.87% with feature-matched distractors and further to 70.96% when coordinate ticks are removed. This two-tier evaluation design enables TangramPuzzle to assess foundational spatial reasoning across diverse model capability levels while providing a more demanding evaluation for stronger models.

2. Task 2: Distinguishing Advanced Geometric Assembly Capabilities. We evaluated both models on all 668 Task 2 instances, obtaining the following results:

Evaluation results of recent MLLMs on Task 2.

ModelTSE↓RGE↓PE↓VPR↑IoU↑Hausdorff↓Success↑
GPT-6.1-sol1.801.6555.2442.3798.390.020542.22
Qwen3.8-27B1.2084.2892.374.4968.840.55864.19

These results demonstrate TangramPuzzle's ability to differentiate recent MLLMs through both strict success criteria and fine-grained geometric diagnostics. GPT-6.1-sol achieves 42.22% Success, substantially outperforming Qwen3.8-27B (4.19%), while Qwen3.8-27B's high RGE (84.28%) and PE (92.37%) reveal substantial difficulties in preserving piece rigidity and satisfying physical constraints. Despite improvements in recent models, reliable geometric assembly remains challenging, underscoring the continued value of TangramPuzzle as an evaluation benchmark.

Overall, these additional experiments show that Task 1 provides a useful foundational assessment across model capability levels, while Task 2 continues to distinguish stronger models and expose their remaining geometric reasoning limitations. We will incorporate these new evaluations and the enhanced Task 1 variant into the revised manuscript.

Reviewer_d4oLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TangramPuzzle, a benchmark for compositional spatial reasoning in MLLMs. It contains 668 tangram configurations filtered and re-annotated from KiloGram, each encoded in an exact symbolic representation (TCE) that writes coordinates as LaTeX expressions (e.g., \sqrt{2}). There are two tasks:

Task 1 (part-to-whole): given the exact TCE coordinates of the seven pieces, pick the matching silhouette from four options. The three distractors are sampled at random from the rest of the dataset. Task 2 (whole-to-part): given a target silhouette with annotated vertices, output a complete assembly as structured JSON.

Task 2 is scored by a hierarchical verifier. It first checks syntax (TSE), rigid-piece preservation (RGE), and physical constraints (PE = overlap OE ∨ connectivity CE). It then measures silhouette fidelity (IoU, Hausdorff) and finally Success. Because the verifier compares the union of the predicted pieces against the target rather than against one reference solution, it is intended to accept alternative valid solutions.

About nine proprietary and open MLLMs are evaluated. Gemini3-Pro reaches 98.65% on Task 1 and 21.56% Success on Task 2; every other model is close to 0% Success. Ablations cover ICL, CoT, decomposition prompting, visual-centric and text-only inputs, and a simplified TCE-Light representation. The paper also reports an OE/CE breakdown, a cross-task correlation analysis, and a small human study with three experts. The headline conclusion is that MLLMs tend to match the target silhouette while violating rigid-body and non-overlap constraints ("visual cheating").

Critical strengths

S1. A generative, machine-verifiable, multi-solution-aware task. Most spatial-reasoning benchmarks are discriminative VQA, which can be gamed with language priors. Task 2 instead asks for an exact assembly that is checked programmatically for rigidity, overlap, connectivity, and silhouette fidelity, and it accepts any valid arrangement rather than a single reference (Sec. 3.3). This formulation is the main reason the benchmark could become a lasting community resource rather than another VQA set, so it carries the most weight in favor of acceptance.

S2. The benchmark is far from saturated and includes useful diagnostics. Task 1 accuracy ranges from near chance to 98.65%. On Task 2 only one model achieves non-trivial Success (Table 3), so there is plenty of room to track progress. The OE/CE breakdown (Table 8) indicates that global connectivity, rather than overlap, is the main failure mode. This is one of the cleaner findings in the paper because it does not depend on the RGE tolerance problem discussed below. Releasing all prompts (Listings 1–7) also adds to the paper's value.

Critical weaknesses

W1. The verifier is under-specified and internally inconsistent, so the headline "visual cheating" claim may be a measurement artifact.

  • Conflicting RGE definitions. Sec. 3.3 says RGE compares area, perimeter, and the complete pairwise vertex-distance matrix. App. E.2 compares only area and perimeter, which is not sufficient to establish polygon congruence.
  • Missing thresholds. The paper gives no numerical tolerance for RGE, OE, or CE. It does not define the Success criterion (is IoU = 1 required? what Hausdorff threshold?). It does not say whether two pieces touching only at a vertex count as connected.
  • The data pattern fits a precision artifact better than deliberate distortion. Claude-Sonnet-4.5 has the lowest syntax-error rate (TSE 0.60%) yet an RGE of 98.20%. Qwen3-VL-8B, which per App. K mostly lines up the input pieces, has an RGE of only 40.42%.
  • The GT self-check is circular. Each silhouette is defined as the union of its GT pieces, so a 100% GT pass rate cannot show that the verifier accepts alternative valid solutions or rejects near-misses.

W2. Weaker supporting evidence.

  • Human baseline. Only three experts took part in Task 2, with no reported item count, interface (was the snapping annotation tool available?), or independence from the authors. Their scores vary widely (100 / 72 / 46).
  • Human evaluation reliability. Table 6 reports the SD of rater means as "inter-rater variability", which is not an agreement measure.
Questions for authors
  1. What exact tolerances are used for RGE, OE, and CE, and what defines Success (IoU / Hausdorff thresholds)? Which RGE criterion is implemented (area + perimeter, or the full distance matrix)? For Listing 1 outputs, is RGE computed from the float coords or the symbolic field? Is vertex-only contact counted as connected?
  2. For each model, what were the temperature, max output tokens, reasoning-effort or thinking setting, API version, and access date? What fraction of TSE comes from truncation? How do GPT-5.2 and Claude-Sonnet-4.5 perform with high reasoning effort or extended thinking?
  3. How many of Gemini3-Pro's 144 successes differ from the GT arrangement? Has the verifier been tested on solver-generated alternative solutions and on controlled perturbations (vertex shift δ, overlap area δ)?
  4. How many items did each expert solve? Did they write JSON coordinates or use the snapping annotation tool? Please replace the SD of rater means in Table 6 with ICC .
4 comments

We thank Reviewer d4oL for the careful reading and constructive comments. We address the concerns below.

W1: Regarding the Concern That High RGE May Be a Numerical Precision Artifact.

We thank the reviewer for this observation. We would like to clarify that the observed RGE differences should not be attributed solely to numerical precision artifacts, for the following reasons:

1. Distinct evaluation metrics. TSE and RGE measure fundamentally different properties. TSE evaluates whether the generated output complies with the required syntax and formatting rules, whereas RGE assesses whether the generated pieces preserve their canonical rigid geometry. Therefore, a low TSE does not necessarily imply a low RGE.

2. Numerical robustness. Our evaluation does not require exact floating-point equality. TangramPuzzle employs precise symbolic coordinates, including fractions and radicals, together with an absolute geometric tolerance of 0.01 for rigidity checks. This design accommodates small numerical deviations while detecting violations of rigid-piece geometry.

3. Different Failure Patterns. As illustrated in Figure 4, Qwen3-VL-8B tends to arrange the seven pieces in a row, preserving piece geometry more frequently (RGE 40.42%) but poorly matching the target (IoU 20.25%). In contrast, Claude-Sonnet-4.5 approximates the target silhouette with visibly distorted pieces (RGE 98.20%). These cases illustrate two distinct failure modes: preserving piece rigidity without correct assembly versus approximating the silhouette while deforming individual pieces.

4. Manual Verification. We also manually inspected representative model predictions, including the cases shown in Figure 4. These inspections confirmed visible geometric deformations in the examined outputs, such as altered piece shapes and distorted boundaries, rather than merely imperceptible floating-point discrepancies. This provides additional qualitative evidence that the observed RGE violations are not solely attributable to numerical precision artifacts.

Q1: What exact tolerances are used for RGE, OE, and CE, and what defines Success (IoU / Hausdorff thresholds)? Which RGE criterion is implemented (area + perimeter, or the full distance matrix)? For Listing 1 outputs, is RGE computed from the float coords or the symbolic field? Is vertex-only contact counted as connected?

We thank the reviewer for highlighting the need for a more precise specification of our verifier. We apologize for the documentation omission in Appendix E.2. The actual verifier follows Section 3.3, consistently checking three complementary geometric properties for each piece: area, perimeter, and the complete pairwise vertex-distance matrix. This ensures that each predicted polygon remains geometrically congruent to its canonical counterpart within the specified tolerances. The distance-matrix check was inadvertently omitted from Appendix E.2 but was implemented consistently across all experiments.

TangramPuzzle employs exact symbolic geometric representations, including fractions and radicals expressed in LaTeX, to preserve geometric precision. During verification, coordinates are converted into numerical representations with predefined tolerances to accommodate minor numerical deviations. Specifically:

  • Rigid Geometry Error (RGE): An absolute tolerance of 0.01 is applied independently to area, perimeter, and the maximum deviation in the complete pairwise vertex-distance matrix. An RGE is reported if any criterion exceeds its tolerance.
  • Overlap Error (OE): An overlap error is reported when the sum of pairwise intersection areas among all pieces exceeds 0.01.
  • Connectivity Error (CE): All pieces must form a single connected component, with both edge contact and vertex-only contact considered valid connections.
  • Success: A prediction must pass all syntax, rigidity, and physical feasibility checks, while satisfying IoU > 0.95 and Hausdorff distance < 0.1.

Regarding Listing 1, the displayed coords/symbolic format was a documentation error. All full-TCE experiments used the same unified symbolic-coordinate output schema and evaluation pipeline, with TCE-Light intentionally adopting a different, transformation-based representation. As also illustrated in Figure 2, the actual schema encodes piece coordinates as LaTeX strings. A simplified excerpt is provided below:

{
  "final_state": {
    "pieces": [{
      "piece_id": "LT_1",
      "piece_type": "Large_Triangle",
      "vertices": [
        {"coords": ["\\frac{907}{768}+\\frac{5\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{\\sqrt{2}}{3}"]}
      ],
      "edges": [{
        "coords": [
          ["\\frac{907}{768}+\\frac{5\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{\\sqrt{2}}{3}"],
          ["\\frac{907}{768}+\\frac{8\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{4\\sqrt{2}}{3}"]
        ],
        "length": "2"
      }]
    }]
  }
}

The verifier reads the symbolic coordinate expressions from vertices[*].coords rather than using a separate float-coordinate field. These expressions are converted into numerical coordinates for geometric evaluation. We will correct Appendix G accordingly and revise Appendix E.2 to document the complete rigidity checks, numerical tolerances, connectivity definition, and success criteria.

Q2: For each model, what were the temperature, max output tokens, reasoning-effort or thinking setting, API version, and access date? What fraction of TSE comes from truncation? How do GPT-5.2 and Claude-Sonnet-4.5 perform with high reasoning effort or extended thinking?

We thank the reviewer for requesting additional implementation details and reasoning-effort comparisons. We clarify the following:

1. Inference Settings and Model Versions. To clarify the “default hyperparameters” description in the original manuscript, we used a common decoding configuration with temperature = 0 and a requested maximum output-token budget of 65,536, subject to the limits supported by each inference interface. GPT-5.2 used reasoning_effort=none, Claude-Sonnet-4.5 used its default setting without Extended Thinking, and Qwen3-VL (Instruct), InternVL3, and GLM-4.6V were evaluated in non-thinking modes. Gemini3-Pro used its default high thinking setting. The API model versions, access dates, and truncation-related proportions of TSE are summarized below. We will revise the original hyperparameter description and document these settings more explicitly in the revised manuscript.

Inference settings and truncation statistics.

ModelAPI VersionAccess DateTruncation / TSE
Qwen3-VL-8Bqwen3-vl-8b-instruct2026-03-0740.4%
Qwen3-VL-32Bqwen3-vl-32b-instruct2026-03-0713.2%
InternVL3-78Bopengvlab/internvl3-78b2025-12-2913.6%
GLM-4.6Vzai-org/GLM-4.6V2026-01-012.4%
GPT-5.2gpt-5.22026-03-064.2%
Gemini3-Progemini-3-pro-preview2026-03-070.0%
Claude-Sonnet-4.5claude-sonnet-4-5-202509292025-12-280.0%

2. Truncation Analysis. Truncation accounts for only a small fraction of TSE in most models and is therefore not the primary source of their syntax failures. The main exception is Qwen3-VL-8B, for which 63 of 156 TSE cases (40.4%) exhibit strong truncation indicators. Inspection of these outputs revealed repeated reasoning passages and recurring attempts to revise unsuccessful geometric configurations, sometimes ending before a valid final response was produced. This suggests that Qwen3-VL-8B's elevated TSE is partly associated with repetitive, unsuccessful reasoning on challenging instances rather than formatting errors alone.

3. Higher Reasoning Effort. In response to the reviewer's suggestion, we conducted additional experiments on 50 randomly selected Task 2 instances, comparing the original settings of GPT-5.2 and Claude-Sonnet-4.5 with reasoning_effort=high and Extended Thinking, respectively. GPT-5.2 achieves 34.0% Success with high reasoning effort, compared with 0.0% under its original setting, whereas Claude-Sonnet-4.5 shows no improvement in Success with Extended Thinking. These findings highlight TangramPuzzle's ability to distinguish model capabilities under different reasoning configurations and suggest that stronger reasoning can substantially improve geometric assembly for some models, but its benefits are not universal. We will include this additional comparison and clarify the reasoning configurations in the revised manuscript.

Higher reasoning effort on 50 Task 2 instances.

Model / SettingTSE↓RGE↓PE↓VPR↑IoU↑Hausdorff↓Success↑
Claude-Sonnet-4.5 (Default)2.096.098.00.066.10.5120.0
Claude-Sonnet-4.5 (Extended Thinking)0.098.0100.00.062.70.6200.0
GPT-5.2 (None)4.090.096.00.056.40.8030.0
GPT-5.2 (High)4.04.064.034.095.60.16034.0

Q3: How many of Gemini3-Pro's 144 successes differ from the GT arrangement? Has the verifier been tested on solver-generated alternative solutions and on controlled perturbations (vertex shift δ, overlap area δ)?

We thank the reviewer for raising this important question regarding the verifier's ability to accept alternative valid solutions and detect geometric violations. We clarify the following:

1. Alternative Valid Solutions. Among Gemini3-Pro's 144 successful predictions, 112 (77.8%) differ from the GT arrangement, excluding permutations of identical piece IDs. Specifically, 17 (11.8%) are globally symmetry-equivalent to the GT through rotation or reflection, while 95 (66.0%) represent distinct internal arrangements beyond global symmetry equivalence, demonstrating the importance of multi-solution-aware evaluation. Such alternative assemblies can arise from different local decompositions, where distinct combinations of tangram pieces form the same subregion. We also tested the verifier on independently solver-generated alternative valid solutions, which were successfully accepted without requiring agreement with the GT internal arrangement.

2. Controlled Perturbation Tests. To further validate the verifier's robustness, we conducted controlled perturbation experiments on valid GT configurations, examining both vertex displacement and piece overlap. For vertex perturbations, we randomly selected one vertex of LT_1 and displaced it in a random direction by magnitude δ, conducting 500 trials per setting. Importantly, vertex displacement does not directly correspond to the resulting area, perimeter, or pairwise vertex-distance deviations, which are checked independently against the 0.01 tolerance. For overlap perturbations, we rigidly translated LT_2 toward LT_1 and used binary search to obtain specified total pairwise overlap areas.

Controlled perturbation results.

Controlled PerturbationVerifier Result
Vertex shift: δ = 0, 0.001100% RGE Pass; 100% Success
Vertex shift: δ = 0.0155.0% RGE Fail; 45.0% Success
Vertex shift: δ = 0.05, 0.1100% RGE Fail; 0% Success
Total overlap: δ = 0, 0.005, 0.01OE Pass; Success
Total overlap: δ = 0.02, 0.05OE Fail; No Success

These results demonstrate that the verifier tolerates minor geometric deviations while reliably detecting violations of the prescribed constraints. At δ = 0.01, some vertex perturbations exceed the geometric tolerance because the induced area or perimeter deviation can be larger than the displacement itself; the resulting mixture of passes and failures is therefore consistent with the defined RGE criteria. Larger vertex perturbations (δ = 0.05 and 0.1) are consistently rejected, while the overlap tests produce the expected outcomes around the 0.01 overlap-area threshold, further supporting the reliability of the verifier's geometric constraint checks.

3. Purpose of GT Self-Check. We agree that GT self-validation alone is insufficient to establish verifier reliability. Its primary purpose is to confirm instance solvability and annotation consistency, ensuring that every benchmark instance has at least one valid solution. The additional alternative-solution and controlled-perturbation tests provide independent evidence that the verifier can accept valid non-GT assemblies and reject geometrically invalid predictions. We will clarify these distinctions and include the supplementary validation results in the revised manuscript

Q4: How many items did each expert solve? Did they write JSON coordinates or use the snapping annotation tool? Please replace the SD of rater means in Table 6 with ICC.

We thank the reviewer for highlighting the need for clearer human evaluation protocols and a more appropriate measure of inter-rater reliability. We clarify the following:

1. Human Baseline Protocol. As described in Section 4.3 and Appendix J, each of the three experts attempted 50 Task 2 instances using our self-developed interactive annotation tool with vertex- and edge-snapping, rather than manually writing JSON coordinates. The three experts were independent of the authors, were not involved in dataset construction or annotation, and had no prior access to the reference solutions. We have further analyzed the differences in their success rates (100%, 72%, and 46%) in Appendix J, discussing their distinct solving strategies. Specifically, systematic solvers decomposed the target silhouette into geometric subregions before placing pieces, whereas trial-and-error solvers relied more heavily on iterative placement and adjustment. We will make these experimental details more explicit in the revised manuscript.

2. Inter-Rater Reliability. We agree that the standard deviation of rater means in Table 6 does not directly measure inter-rater agreement. Following the reviewer's suggestion, we computed ICC(2,3) from the experts' item-level ratings, using a two-way random-effects model with absolute agreement. The model-specific ICC values range from 0.80 to 0.91, indicating good to excellent reliability across the three evaluation dimensions. We will replace the reported standard deviations with these ICC values in the revised Table 6.

Table 6: Human evaluation of MLLMs on Task 2.

ExpertPSM (↑)PPC (↑)NMC (↑)
Gemini3-Pro
Expert 14.204.304.60
Expert 24.504.604.10
Expert 34.204.504.00
Average4.304.474.23
ICC(2,3)0.880.910.80
GPT-5.2
Expert 11.601.501.80
Expert 21.201.101.10
Expert 31.501.201.60
Average1.431.271.50
ICC(2,3)0.850.890.83

ICC(2,3) denotes the two-way random-effects, absolute-agreement intraclass correlation coefficient for the average ratings of three experts, computed separately for each model and evaluation dimension.

Reviewer_EodBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TangramPuzzle evaluates compositional spatial reasoning using 668 configurations and 1,336 instances across outline prediction and constrained assembly generation. Its main contributions are the TCE representation and a verifier separating geometric validity from silhouette fidelity. Model comparisons, ablations, ground-truth checks, and human evaluations reveal a substantial gap between plausible silhouettes and valid assemblies.

Critical strengths

The evaluation distinguishes visual similarity from geometric correctness while accepting multiple valid solutions. The annotated dataset, ground-truth checks, and human comparisons support its value as a benchmark.

Critical weaknesses
  1. The verifier is inconsistently specified: Section 3.3 includes vertex-distance checks, whereas Appendix E.2 describes only area and perimeter. Success criteria and tolerances are also unclear.
  2. Appendix G changes output schemas across ablations, confounding their interpretation. The baseline already requests step-by-step reasoning, further obscuring the CoT comparison.
Questions for authors
  1. Which rigidity checks, success criteria, and tolerances were implemented?
  2. Are the schema differences documentation errors? If not, do the ablation conclusions hold with a fixed output schema and comparable inference budgets?
2 comments

We thank Reviewer EodB for the insightful comments and the recognition of our work. Below are our point-to-point responses to the concerns raised:

Q1: Which rigidity checks, success criteria, and tolerances were implemented?

We thank the reviewer for pointing out the inconsistency between Section 3.3 and Appendix E.2. We apologize for this documentation omission. The actual verifier implementation follows Section 3.3, consistently checking three complementary geometric properties for each piece: area, perimeter, and the complete pairwise vertex-distance matrix. This ensures that each predicted polygon remains geometrically congruent to its canonical counterpart within the specified tolerances. The distance-matrix check was inadvertently omitted from Appendix E.2 but was implemented consistently across all experiments.

To clarify the numerical tolerances and evaluation procedure, TangramPuzzle employs exact symbolic geometric representations, including fractions and radicals expressed in LaTeX, to preserve geometric precision. During verification, these expressions are converted into numerical geometric representations, and predefined tolerances are applied to accommodate minor numerical approximations. Specifically:

  • Rigid Geometry Error (RGE): An absolute tolerance of 0.01 is applied independently to area, perimeter, and the maximum deviation in the complete pairwise vertex-distance matrix. An RGE is reported if any of these three criteria exceeds its tolerance.
  • Physical Error (PE): An overlap error is reported when the sum of pairwise intersection areas among all pieces exceeds 0.01. Connectivity requires the assembled pieces to form a single connected component, with both edge contact and vertex-only contact considered valid connections.
  • Success: A prediction must pass all syntax, rigidity, and physical feasibility checks, while simultaneously satisfying IoU > 0.95 and Hausdorff distance < 0.1 with respect to the target silhouette.

Importantly, Success is determined by both geometric validity and silhouette fidelity, rather than silhouette similarity alone. Thus, a prediction cannot be considered successful merely by achieving high IoU if it violates rigid-piece or physical constraints. We will revise Appendix E.2 to explicitly include the complete vertex-distance matrix check, numerical tolerances, connectivity definition, and success criteria, ensuring full consistency with Section 3.3 and the implemented verifier.

Q2: Are the schema differences documentation errors? If not, do the ablation conclusions hold with a fixed output schema and comparable inference budgets?

We thank the reviewer for carefully examining the prompt templates in Appendix G and highlighting the potential confounding factors in our ablation studies. We would like to clarify the following:

1. Output Schema Consistency. The schema differences in Appendix G are documentation errors introduced during manuscript preparation, rather than differences in the actual experimental implementation. All full-TCE prompting strategies, including Vanilla, ICL, CoT, and decomposition-based prompting, used the same unified symbolic-coordinate output schema, parsing pipeline, and evaluation criteria. TCE-Light is an intentional exception, as it uses a different, transformation-based output representation. As illustrated in Figure 2, the unified schema represents piece geometry using symbolic coordinates expressed as LaTeX strings. A simplified excerpt is shown below:

{
  "final_state": {
    "pieces": [{
      "piece_id": "LT_1",
      "piece_type": "Large_Triangle",
      "vertices": [
        {"coords": ["\\frac{907}{768}+\\frac{5\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{\\sqrt{2}}{3}"]}
      ],
      "edges": [{
        "coords": [
          ["\\frac{907}{768}+\\frac{5\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{\\sqrt{2}}{3}"],
          ["\\frac{907}{768}+\\frac{8\\sqrt{2}}{3}", "-\\frac{361}{768}+\\frac{4\\sqrt{2}}{3}"]
        ],
        "length": "2"
      }]
    }]
  }
}

The verifier reads the LaTeX coordinate expressions from vertices[*].coords, converts them into numerical coordinates, and evaluates geometric rigidity, physical feasibility, and silhouette fidelity. Therefore, the reported ablation results were not confounded by differences in output formats. We will correct the corresponding listings in Appendix G to ensure consistency with the actual implementation.

2. Clarification of the CoT Comparison. We agree that the Vanilla baseline already encourages general step-by-step reasoning. Thus, our original CoT ablation was intended to investigate whether explicit geometry-specific structured reasoning provides additional benefits over general step-by-step reasoning, rather than comparing reasoning versus no reasoning.

3. Inference Budget Consistency. The compared prompting strategies were evaluated using consistent inference configurations, including the same output-token limits and decoding settings for each model. Although different prompts may result in different actual token consumption, the comparisons were conducted under the same configured inference budgets.

4. Additional No-CoT Experiment. To further address the reviewer's concern, we conducted an additional full-set experiment on Qwen3-VL-32B-Instruct using a Direct Answer (No-CoT) setting. Specifically, we replaced the original instruction to provide step-by-step spatial reasoning before the final JSON output with an instruction to output only the final JSON, without any reasoning text. The remaining task inputs, output schema, evaluation procedure, and inference configuration were kept unchanged.

No-CoT ablation on Qwen3-VL-32B-Instruct (Task 2).

SettingTSE ↓RGE ↓PE ↓VPR ↑IoU ↑Hausdorff ↓Success ↑
Direct Answer5.8489.8297.160.0042.711.670.00
Vanilla7.9383.0894.460.0050.551.390.00
Structured CoT6.2983.8395.510.1551.511.380.15

Removing the step-by-step reasoning instruction decreases IoU from 50.55% to 42.71%, while structured CoT provides only marginal improvements over Vanilla. These results further support our finding in Section 4.3 that although step-by-step reasoning can improve silhouette matching, neither general nor geometry-specific reasoning instructions are sufficient to reliably solve the strict geometric assembly task. We will incorporate this comparison and clarify the CoT ablation in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper 14121: A Gentle Reminder to Reviewers

Dear Reviewers,

Thank you very much for your insightful suggestions and comments. As the discussion phase is drawing to a close, we would like to kindly check if our detailed responses, along with the newly incorporated experiments and revisions, have sufficiently addressed your concerns.

We look forward to providing any further clarifications or addressing any additional questions you might have during the remaining time of the discussion phase. Once again, we sincerely appreciate your time, effort, and invaluable guidance in refining this work.

Best regards,

The Authors of Paper 14121

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.