- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper constructs a new benchmark to evaluate the continuous geometric reasoning ability of vision-language models (VLMs). The benchmark consists of two complementary tasks. TangramPose asks a model to recover the identity and continuous pose (position, rotation, scale) of every visible piece in scenes containing one to seven objects. CanvasPlace asks a model to generate any target pose satisfying geometric constraints such as containment, spatial relations, clearance, orientation, and collision avoidance. The benchmark contains 4,970 Tangram scenes and 1,500 CanvasPlace tasks with controlled difficulty tiers. Across ten proprietary and open VLMs, the paper finds that most models produce parseable outputs and non-trivial scene-level overlap, but fail strict object-level reconstruction. The paper then compares in-context demonstrations, a structured head over frozen visual features (M1), language-only LoRA (M2), and additional visual and cross-modal adaptation (M3). Probing, controlled counterfactual edits, and activation interventions are utilized to diagnose the gap between representation and prediction.
Critical strengths
- The paper focuses on valuable continuous geometric reasoning tasks and designs complementary evaluation tasks. TangramPose probes recovery of an observed configuration, while CanvasPlace probes construction of a new constraint-satisfying configuration. Together they distinguish reading geometry from using geometry, which is a useful conceptual distinction.
- The paper conducts extensive experimental evaluations on 10 models, including both proprietary and open VLMs.
Critical weaknesses
- The paper lacks input, output, and answer examples for two tasks, TangramPose and CanvasPlace. Many evaluation metrics have been designed, but the relationship between these metrics and model capabilities has not been fully analyzed. Therefore, it is difficult to determine the superiority of the constructed benchmark over existing benchmarks.
- Section 4.1 and Table 1 specify 200 Tangram scenes per piece count, whereas Appendix A.3 and Table 8 specify 20. CanvasPlace is described as having 500 source families and evaluating 200, but Appendix B also refers to a complete 50-family bank, and Tables 9–10 explicitly evaluate 20 families. Benchmark accounting and reported aggregates are internally inconsistent.
- On TangramPose, eight of ten models achieve zero Joint success, and Claude reaches only 4.29%. The benchmark may be too hard for current models, which weakens its ability to discriminate among them. A more graded difficulty ladder or partial-credit metrics in the main setting would improve utility.
- The generalization experiments are informative but underexplained. The angle-OOD result is dramatic: M3 angle error rises from 1.63° to 101.05° and Joint drops from 70.24% to 3.33%. This is a severe failure, but the paper does not diagnose why.
Questions for authors
- Is the benchmark appropriately calibrated? Eight of ten models achieve zero Joint success on TangramPose. Does this reflect a genuine capability gap, or does the strict Joint criterion make the benchmark too hard to discriminate among models? Can you report partial-credit or graded metrics in the paper?
- Can you provide a compact summary table of all metrics? The paper introduces many metrics: position error, angle error, scale error, Piece IoU, Scene IoU, Joint success, full-polygon containment, centroid based spatial relations, polygon intersections, shortest edge-to-edge distances, and symmetry equivalent orientations A single table mapping each task to its metrics and main result would greatly improve readability.
8 comments
W1, Q2. What the model sees, predicts, and is scored on (continued)
Table 16. What each benchmark metric measures.
| Metric | What the evaluator checks |
|---|---|
| TangramPose — recover each identified piece | |
| Parseable output | Response yields a readable list of pieces. |
| Exact IDs / piece recall | Correct count and full set of IDs per scene / fraction of true pieces matched by ID. |
| Position error | Euclidean distance between predicted and true centroids on the 0–10 canvas. |
| Angle error | Rotation error accounting for the symmetry period (360° for triangles, 90° for squares, 180° for parallelograms). |
| Relative scale error | Absolute scale difference divided by true scale. |
| Attribute success | Fraction of all target pieces within the position (0.5), angle (15°), or scale (10%) tolerance, scored separately. |
| Reflection accuracy | Correct flip for a uniquely matched parallelogram. |
| Piece IoU / Scene IoU | Intersection-over-union of predicted and true piece masks matched by ID / of the two scene unions. |
| Scene IoU ≥ 0.8 | Fraction of scenes whose union overlap reaches 0.8. |
| Complete pose / Joint | Position, angle, and scale pass together, with the correct parallelogram flip / exact IDs and every piece passing these checks. |
| CanvasPlace — place the target subject to the instruction | |
| API response / parsing | Request returns an answer / answer yields finite coordinates and angle, with coordinates in [0,1]. |
| Containment | The whole transformed polygon lies inside the tray. |
| Spatial relations | Centroids satisfy strict left, right, above, below, or horizontal-between relations. |
| Clearance | Shortest edge-to-edge distance meets the requested minimum. |
| Task orientation | Angle matches an allowed target orientation, accounting for symmetry, within 1°. |
| Cardinal rotation | Angle is within 1° of a multiple of 90°. |
| Collision avoidance | No positive-area intersection with any solid object. |
| Strict success | All applicable geometric checks pass on the same task. |
| Strict given valid | Strict success among parseable poses. |
TangramPose errors and Piece IoU use ID-matched pieces; triangle reflections are absorbed into the effective angle. Per-target success counts missing, malformed, or duplicate-ID targets as failures. CanvasPlace constraint rates pool applicable constraint instances, counting invalid outputs as failures; polygon checks use numerical tolerance 10⁻⁷.
W2. Dataset size and evaluation size
How does the number of source scenes or families relate to the samples used for evaluation?
W2 Response. Many thanks for the reviewer providing this comment. The 20 samples per group came from an earlier smoke test. Our data-generation scripts can produce as many TangramPose scenes and CanvasPlace families as needed. In the reported evaluation, TangramPose has 200 scenes for each piece count from one through seven, generated from 14 held-out source pages. Every one of the ten models is evaluated on all 1,400 scenes (200 × 7), yielding 14,000 model–scene predictions in total. CanvasPlace has 200 evaluated scene families, 20 for each of ten target shapes. Each family is evaluated at Easy, Medium and Hard difficulty for every model: 200 × 3 = 600 tasks per model, or 6,000 model–task predictions across ten models. Section 4.1, Table 1 and the appendices use these evaluation denominators consistently.
Revised manuscript: Introduction, lines 43–44; Section 3.1.2, lines 135–140; Section 4.1, lines 224–228; and Appendix B.1, lines 1141–1146.
W3, Q1. Partial credit and sensitivity to scoring tolerances
When whole-scene recovery is rare, can TangramPose distinguish models? How do per-piece scores and relaxed scoring tolerances change the comparison?
Response. We thank the reviewer making these comments. Whole-scene Joint alone separates the weaker models poorly: eight of the ten score between 0.00% and 0.57%. Scoring each piece also shows what a model gets right. For example, GPT-4o and Qwen3-VL-30B-A3B both score 0.00% Joint, yet 22.73% of Qwen’s pieces meet the scale tolerance versus 4.96% for GPT-4o. Table 17 scores position, angle, scale, and complete pose over 5,600 target pieces per model; Joint requires every piece in a scene to pass. Revised Appendix A, Table 8, also reports continuous errors and overlap at each piece count from one to seven.
Updated continuous errors and overlaps are in revised Appendix A.3, Table 8, lines 810–879. The new partial-credit and tolerance analyses are in Tables 17–18 of this response.
Table 17. Partial credit and complete reconstruction (%)
| Model | Position | Angle | Scale | Complete pose | Scene Joint |
|---|---|---|---|---|---|
| GPT-6 Astra | 99.80 | 98.04 | 99.16 | 97.46 | 90.86 |
| GPT-5.6 Luna | 54.14 | 19.18 | 5.30 | 0.64 | 0.14 |
| GPT-4o | 9.64 | 15.00 | 4.96 | 0.05 | 0.00 |
| Gemini 3.1 Pro | 55.86 | 26.71 | 8.88 | 1.96 | 0.57 |
| Gemini Flash Lite | 39.04 | 23.41 | 7.50 | 0.70 | 0.07 |
| Claude Sonnet 4.6 | 46.18 | 19.07 | 11.68 | 2.05 | 2.79 |
| Qwen3-VL-8B | 2.18 | 11.86 | 17.45 | 0.05 | 0.00 |
| Qwen3-VL-30B-A3B | 10.73 | 13.93 | 22.73 | 0.43 | 0.00 |
| Gemma-4-31B | 40.11 | 20.91 | 4.68 | 0.55 | 0.43 |
| Llama-4 Maverick | 9.82 | 14.66 | 11.30 | 0.18 | 0.07 |
Position, angle, scale and complete pose use all 5,600 target pieces per model. Missing or duplicate matches fail the object check. Scene Joint uses all 1,400 scenes.
We rescored the saved predictions with relaxed continuous tolerances while retaining the same identity, piece-count and reflection checks. Doubling the position, angle and scale tolerances raised Joint from 90.86% to 92.57% for GPT-6 and from 0.57% to 1.43% for Gemini 3.1 Pro. At four times the original tolerances the rates were 93.64% and 4.21%. Threshold choice affects absolute success, but the performance gap persists. Table 18 reports all five tolerance settings.
Table 18. Joint success under alternative pose tolerances (%)
| Model | 0.25× | 0.5× | 1× | 2× | 4× |
|---|---|---|---|---|---|
| GPT-6 Astra | 85.00 | 88.43 | 90.86 | 92.57 | 93.64 |
| GPT-5.6 Luna | 0.00 | 0.00 | 0.14 | 0.71 | 3.93 |
| GPT-4o | 0.00 | 0.00 | 0.00 | 0.21 | 3.36 |
| Gemini 3.1 Pro | 0.14 | 0.21 | 0.57 | 1.43 | 4.21 |
| Gemini Flash Lite | 0.00 | 0.00 | 0.07 | 0.64 | 4.00 |
| Claude Sonnet 4.6 | 0.57 | 1.86 | 2.79 | 3.79 | 7.00 |
| Qwen3-VL-8B | 0.00 | 0.00 | 0.00 | 0.07 | 1.57 |
| Qwen3-VL-30B-A3B | 0.00 | 0.00 | 0.00 | 0.43 | 4.36 |
| Gemma-4-31B | 0.00 | 0.00 | 0.43 | 0.93 | 2.57 |
| Llama-4 Maverick | 0.00 | 0.00 | 0.07 | 0.36 | 3.43 |
We multiply the position, angle and scale tolerances together. Identity, count and reflection requirements are fixed. The original thresholds reproduce Table 1.
W1, Q2. What the model sees, predicts, and is scored on
For each task, what are the inputs, prediction and correct answer? Can one table show what each metric measures and what the results reveal beyond spatial relations or scene overlap?
Response. Thank you very much for providing this comment. TangramPose tests whether a model can recover an observed arrangement. Given one image of colored pieces and a key identifying them, the model returns each piece’s ID, center, angle and scale, plus the flip for the parallelogram if present. We compare each prediction with that piece’s annotated pose. In a five-piece scene, GPT-6 placed the parallelogram’s center at (2.422, 6.497); its true center was (2.419, 6.498). All five pieces passed the pose checks, so the scene passed Joint (Scene IoU 0.981).
CanvasPlace tests whether a model can construct a valid placement. It sees a scene, a separate image of the piece to place, and a written instruction. It predicts the center and rotation of that piece. Any pose satisfying all constraints is correct. In Task 43, the U-shaped piece satisfied the requested spatial relations but overlapped the green triangle by 0.13 square pixels, so it failed. The revised paper also shows API pass/fail examples for GPT-6 Astra (Appendix A.4, Figure 8) and Gemini 3.1 Pro (Appendix B.3, Figure 12). Table 16 summarizes the metrics; we have added the same definitions to the revised paper (Appendix H, Table 43).
Added worked examples in the revised manuscript: Appendix A.4, Figure 8, lines 1093–1096; Appendix B.3, Figure 12, lines 1473–1476; Appendix D.2, Figure 13, lines 1770–1773; Appendix E.3, Figure 14, lines 1965–1969. Updated constraint results: Appendix B.2, Table 9, lines 1218–1226. Added metric definitions: Appendix H, Table 43, lines 2595–2624.
Section 4.3, Figure 3 shows five-piece reconstructions for Qwen and InternVL, and Table 3 reports aggregate results for the same checkpoints. Appendix D.2, Figure 13 in the revised paper pairs a Joint pass and failure for each M3 model.
Examples 1 and 2 give a passing and a failing Qwen M3 prediction from the aligned transfer experiment. Appendix E.3, Figure 14 in the revised paper shows the scene, target and predicted placement for each. The raw answers use a 0–10 grid with the origin at bottom left and counterclockwise angles. We also give the same poses in the evaluator’s coordinates (0–1, origin at top left, clockwise angles). The reference pose is one feasible answer.
Example 1. task_08_square_left_of_circle
Instruction: Put the yellow square fully inside the orange tray, to the left of the purple circle, while staying at least 12 pixels from the gray hexagon.
Raw response: {"x":5.2,"y":5.2,"theta":0}. Evaluator pose (x, y, clockwise angle): (0.52, 0.48, 0°). One feasible reference pose is (0.430, 0.469, 0°).
Pass. The yellow square lies entirely inside the orange tray. Its center is 123.76 pixels left of the purple circle’s center, its nearest edge is 46.41 pixels from the gray hexagon (12 required), and it overlaps no solid object.
Example 2. task_01_l_in_tray
Instruction: Place the yellow L-shaped piece completely inside the green tray, to the right of the blue circle, without touching the red square, with its long arm pointing upward.
Raw response: {"x":7.2,"y":5.8,"theta":270}. Evaluator pose (x, y, clockwise angle): (0.72, 0.42, 90°). One feasible reference pose is (0.771, 0.410, 0°).
Fail. The yellow L lies inside the green tray, to the right of the blue circle, clears the red square, and overlaps no solid object. Its long arm is 90° from the required upward orientation. That single error invalidates the placement.
On Hard CanvasPlace tasks, 96.22% of applicable spatial-relation checks passed, but only 29.40% of placements satisfied every constraint (revised paper, Appendix B, Table 9). A target can satisfy spatial relations yet collide with an object or extend beyond the tray. For TangramPose, Scene IoU measures scene-union overlap, whereas Joint requires correct identities and all per-piece pose checks to pass (Table 16).