acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking and Diagnosing Continuous Geometric Reasoning in Vision-Language Models

Abstract

Vision-language models (VLMs) perform well on many spatial benchmarks, but their ability to reason over continuous geometry remains unclear. We introduce two complementary tasks: TangramPose, requiring object-specific prediction of position, rotation, and scale, and CanvasPlace, requiring placements that satisfy geometric constraints. Across 10 VLMs, most models achieve reasonable scene overlap yet fail strict object-level reconstruction, while the strongest model per- forms substantially better in small number piece task but fail when piece number increase. In-context demonstrations mainly improve scale calibration, whereas fine-tuning yields much larger gains, even with a frozen visual encoder. However, these gains transfer poorly to disjoint rotation ranges, unseen piece counts, and a distinct placement task. Probing shows that angle and scale are decodable from the base models’ visual representations far more accurately than they are expressed in final predictions. Counterfactual and activation-patching analyses further show that adaptation improves object-specific responses and strengthens the influence of target-region states on geometric outputs. Together, our results distinguish accurate in-distribution geometric prediction from robust, reusable geometric competence, and provide a framework for analyzing where VLMs succeed and fail in continuous geometric reasoning. Dataset and code will be released upon acceptance

Primary area:
datasets and benchmarks
Keywords:
vision-language models, continuous geometric reasoning, spatial reasoning, evaluation benchmarks, in context learning, parameter-efficient fine-tuning, mechanistic interpretability, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jtbXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs a new benchmark to evaluate the continuous geometric reasoning ability of vision-language models (VLMs). The benchmark consists of two complementary tasks. TangramPose asks a model to recover the identity and continuous pose (position, rotation, scale) of every visible piece in scenes containing one to seven objects. CanvasPlace asks a model to generate any target pose satisfying geometric constraints such as containment, spatial relations, clearance, orientation, and collision avoidance. The benchmark contains 4,970 Tangram scenes and 1,500 CanvasPlace tasks with controlled difficulty tiers. Across ten proprietary and open VLMs, the paper finds that most models produce parseable outputs and non-trivial scene-level overlap, but fail strict object-level reconstruction. The paper then compares in-context demonstrations, a structured head over frozen visual features (M1), language-only LoRA (M2), and additional visual and cross-modal adaptation (M3). Probing, controlled counterfactual edits, and activation interventions are utilized to diagnose the gap between representation and prediction.

Critical strengths
  1. The paper focuses on valuable continuous geometric reasoning tasks and designs complementary evaluation tasks. TangramPose probes recovery of an observed configuration, while CanvasPlace probes construction of a new constraint-satisfying configuration. Together they distinguish reading geometry from using geometry, which is a useful conceptual distinction.
  2. The paper conducts extensive experimental evaluations on 10 models, including both proprietary and open VLMs.
Critical weaknesses
  1. The paper lacks input, output, and answer examples for two tasks, TangramPose and CanvasPlace. Many evaluation metrics have been designed, but the relationship between these metrics and model capabilities has not been fully analyzed. Therefore, it is difficult to determine the superiority of the constructed benchmark over existing benchmarks.
  2. Section 4.1 and Table 1 specify 200 Tangram scenes per piece count, whereas Appendix A.3 and Table 8 specify 20. CanvasPlace is described as having 500 source families and evaluating 200, but Appendix B also refers to a complete 50-family bank, and Tables 9–10 explicitly evaluate 20 families. Benchmark accounting and reported aggregates are internally inconsistent.
  3. On TangramPose, eight of ten models achieve zero Joint success, and Claude reaches only 4.29%. The benchmark may be too hard for current models, which weakens its ability to discriminate among them. A more graded difficulty ladder or partial-credit metrics in the main setting would improve utility.
  4. The generalization experiments are informative but underexplained. The angle-OOD result is dramatic: M3 angle error rises from 1.63° to 101.05° and Joint drops from 70.24% to 3.33%. This is a severe failure, but the paper does not diagnose why.
Questions for authors
  1. Is the benchmark appropriately calibrated? Eight of ten models achieve zero Joint success on TangramPose. Does this reflect a genuine capability gap, or does the strict Joint criterion make the benchmark too hard to discriminate among models? Can you report partial-credit or graded metrics in the paper?
  2. Can you provide a compact summary table of all metrics? The paper introduces many metrics: position error, angle error, scale error, Piece IoU, Scene IoU, Joint success, full-polygon containment, centroid based spatial relations, polygon intersections, shortest edge-to-edge distances, and symmetry equivalent orientations A single table mapping each task to its metrics and main result would greatly improve readability.
8 comments

W1, Q2. What the model sees, predicts, and is scored on

For each task, what are the inputs, prediction and correct answer? Can one table show what each metric measures and what the results reveal beyond spatial relations or scene overlap?

Response. Thank you very much for providing this comment. TangramPose tests whether a model can recover an observed arrangement. Given one image of colored pieces and a key identifying them, the model returns each piece’s ID, center, angle and scale, plus the flip for the parallelogram if present. We compare each prediction with that piece’s annotated pose. In a five-piece scene, GPT-6 placed the parallelogram’s center at (2.422, 6.497); its true center was (2.419, 6.498). All five pieces passed the pose checks, so the scene passed Joint (Scene IoU 0.981).

CanvasPlace tests whether a model can construct a valid placement. It sees a scene, a separate image of the piece to place, and a written instruction. It predicts the center and rotation of that piece. Any pose satisfying all constraints is correct. In Task 43, the U-shaped piece satisfied the requested spatial relations but overlapped the green triangle by 0.13 square pixels, so it failed. The revised paper also shows API pass/fail examples for GPT-6 Astra (Appendix A.4, Figure 8) and Gemini 3.1 Pro (Appendix B.3, Figure 12). Table 16 summarizes the metrics; we have added the same definitions to the revised paper (Appendix H, Table 43).

Added worked examples in the revised manuscript: Appendix A.4, Figure 8, lines 1093–1096; Appendix B.3, Figure 12, lines 1473–1476; Appendix D.2, Figure 13, lines 1770–1773; Appendix E.3, Figure 14, lines 1965–1969. Updated constraint results: Appendix B.2, Table 9, lines 1218–1226. Added metric definitions: Appendix H, Table 43, lines 2595–2624.

Section 4.3, Figure 3 shows five-piece reconstructions for Qwen and InternVL, and Table 3 reports aggregate results for the same checkpoints. Appendix D.2, Figure 13 in the revised paper pairs a Joint pass and failure for each M3 model.

Examples 1 and 2 give a passing and a failing Qwen M3 prediction from the aligned transfer experiment. Appendix E.3, Figure 14 in the revised paper shows the scene, target and predicted placement for each. The raw answers use a 0–10 grid with the origin at bottom left and counterclockwise angles. We also give the same poses in the evaluator’s coordinates (0–1, origin at top left, clockwise angles). The reference pose is one feasible answer.

Example 1. task_08_square_left_of_circle

Instruction: Put the yellow square fully inside the orange tray, to the left of the purple circle, while staying at least 12 pixels from the gray hexagon.

Raw response: {"x":5.2,"y":5.2,"theta":0}. Evaluator pose (x, y, clockwise angle): (0.52, 0.48, 0°). One feasible reference pose is (0.430, 0.469, 0°).

Pass. The yellow square lies entirely inside the orange tray. Its center is 123.76 pixels left of the purple circle’s center, its nearest edge is 46.41 pixels from the gray hexagon (12 required), and it overlaps no solid object.

Example 2. task_01_l_in_tray

Instruction: Place the yellow L-shaped piece completely inside the green tray, to the right of the blue circle, without touching the red square, with its long arm pointing upward.

Raw response: {"x":7.2,"y":5.8,"theta":270}. Evaluator pose (x, y, clockwise angle): (0.72, 0.42, 90°). One feasible reference pose is (0.771, 0.410, 0°).

Fail. The yellow L lies inside the green tray, to the right of the blue circle, clears the red square, and overlaps no solid object. Its long arm is 90° from the required upward orientation. That single error invalidates the placement.

On Hard CanvasPlace tasks, 96.22% of applicable spatial-relation checks passed, but only 29.40% of placements satisfied every constraint (revised paper, Appendix B, Table 9). A target can satisfy spatial relations yet collide with an object or extend beyond the tray. For TangramPose, Scene IoU measures scene-union overlap, whereas Joint requires correct identities and all per-piece pose checks to pass (Table 16).

W1, Q2. What the model sees, predicts, and is scored on (continued)

Table 16. What each benchmark metric measures.

MetricWhat the evaluator checks
TangramPose — recover each identified piece
Parseable outputResponse yields a readable list of pieces.
Exact IDs / piece recallCorrect count and full set of IDs per scene / fraction of true pieces matched by ID.
Position errorEuclidean distance between predicted and true centroids on the 0–10 canvas.
Angle errorRotation error accounting for the symmetry period (360° for triangles, 90° for squares, 180° for parallelograms).
Relative scale errorAbsolute scale difference divided by true scale.
Attribute successFraction of all target pieces within the position (0.5), angle (15°), or scale (10%) tolerance, scored separately.
Reflection accuracyCorrect flip for a uniquely matched parallelogram.
Piece IoU / Scene IoUIntersection-over-union of predicted and true piece masks matched by ID / of the two scene unions.
Scene IoU ≥ 0.8Fraction of scenes whose union overlap reaches 0.8.
Complete pose / JointPosition, angle, and scale pass together, with the correct parallelogram flip / exact IDs and every piece passing these checks.
CanvasPlace — place the target subject to the instruction
API response / parsingRequest returns an answer / answer yields finite coordinates and angle, with coordinates in [0,1].
ContainmentThe whole transformed polygon lies inside the tray.
Spatial relationsCentroids satisfy strict left, right, above, below, or horizontal-between relations.
ClearanceShortest edge-to-edge distance meets the requested minimum.
Task orientationAngle matches an allowed target orientation, accounting for symmetry, within 1°.
Cardinal rotationAngle is within 1° of a multiple of 90°.
Collision avoidanceNo positive-area intersection with any solid object.
Strict successAll applicable geometric checks pass on the same task.
Strict given validStrict success among parseable poses.

TangramPose errors and Piece IoU use ID-matched pieces; triangle reflections are absorbed into the effective angle. Per-target success counts missing, malformed, or duplicate-ID targets as failures. CanvasPlace constraint rates pool applicable constraint instances, counting invalid outputs as failures; polygon checks use numerical tolerance 10⁻⁷.

W2. Dataset size and evaluation size

How does the number of source scenes or families relate to the samples used for evaluation?

W2 Response. Many thanks for the reviewer providing this comment. The 20 samples per group came from an earlier smoke test. Our data-generation scripts can produce as many TangramPose scenes and CanvasPlace families as needed. In the reported evaluation, TangramPose has 200 scenes for each piece count from one through seven, generated from 14 held-out source pages. Every one of the ten models is evaluated on all 1,400 scenes (200 × 7), yielding 14,000 model–scene predictions in total. CanvasPlace has 200 evaluated scene families, 20 for each of ten target shapes. Each family is evaluated at Easy, Medium and Hard difficulty for every model: 200 × 3 = 600 tasks per model, or 6,000 model–task predictions across ten models. Section 4.1, Table 1 and the appendices use these evaluation denominators consistently.

Revised manuscript: Introduction, lines 43–44; Section 3.1.2, lines 135–140; Section 4.1, lines 224–228; and Appendix B.1, lines 1141–1146.

W3, Q1. Partial credit and sensitivity to scoring tolerances

When whole-scene recovery is rare, can TangramPose distinguish models? How do per-piece scores and relaxed scoring tolerances change the comparison?

Response. We thank the reviewer making these comments. Whole-scene Joint alone separates the weaker models poorly: eight of the ten score between 0.00% and 0.57%. Scoring each piece also shows what a model gets right. For example, GPT-4o and Qwen3-VL-30B-A3B both score 0.00% Joint, yet 22.73% of Qwen’s pieces meet the scale tolerance versus 4.96% for GPT-4o. Table 17 scores position, angle, scale, and complete pose over 5,600 target pieces per model; Joint requires every piece in a scene to pass. Revised Appendix A, Table 8, also reports continuous errors and overlap at each piece count from one to seven.

Updated continuous errors and overlaps are in revised Appendix A.3, Table 8, lines 810–879. The new partial-credit and tolerance analyses are in Tables 17–18 of this response.

Table 17. Partial credit and complete reconstruction (%)

ModelPositionAngleScaleComplete poseScene Joint
GPT-6 Astra99.8098.0499.1697.4690.86
GPT-5.6 Luna54.1419.185.300.640.14
GPT-4o9.6415.004.960.050.00
Gemini 3.1 Pro55.8626.718.881.960.57
Gemini Flash Lite39.0423.417.500.700.07
Claude Sonnet 4.646.1819.0711.682.052.79
Qwen3-VL-8B2.1811.8617.450.050.00
Qwen3-VL-30B-A3B10.7313.9322.730.430.00
Gemma-4-31B40.1120.914.680.550.43
Llama-4 Maverick9.8214.6611.300.180.07

Position, angle, scale and complete pose use all 5,600 target pieces per model. Missing or duplicate matches fail the object check. Scene Joint uses all 1,400 scenes.

We rescored the saved predictions with relaxed continuous tolerances while retaining the same identity, piece-count and reflection checks. Doubling the position, angle and scale tolerances raised Joint from 90.86% to 92.57% for GPT-6 and from 0.57% to 1.43% for Gemini 3.1 Pro. At four times the original tolerances the rates were 93.64% and 4.21%. Threshold choice affects absolute success, but the performance gap persists. Table 18 reports all five tolerance settings.

Table 18. Joint success under alternative pose tolerances (%)

Model0.25×0.5×1×2×4×
GPT-6 Astra85.0088.4390.8692.5793.64
GPT-5.6 Luna0.000.000.140.713.93
GPT-4o0.000.000.000.213.36
Gemini 3.1 Pro0.140.210.571.434.21
Gemini Flash Lite0.000.000.070.644.00
Claude Sonnet 4.60.571.862.793.797.00
Qwen3-VL-8B0.000.000.000.071.57
Qwen3-VL-30B-A3B0.000.000.000.434.36
Gemma-4-31B0.000.000.430.932.57
Llama-4 Maverick0.000.000.070.363.43

We multiply the position, angle and scale tolerances together. Identity, count and reflection requirements are fixed. The original thresholds reproduce Table 1.

Reviewer_hEHrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TangramPose for identity-specific continuous pose recovery and CanvasPlace for generating geometrically feasible placements. It benchmarks VLM deployments and compares demonstrations, structured prediction heads, language-only adaptation, and additional visual adaptation. Distribution-shift tests, oracle-assisted probes, blank-image controls, counterfactual edits, and activation patching investigate the gap between decodable geometric information and reliable autonomous prediction. The diagnostic program is potentially valuable, but inconsistencies in evaluation counts and aggregate results currently prevent a dependable interpretation of the headline benchmark evidence.

Critical strengths
  1. Complementary tasks with appropriate geometric evaluators. TangramPose preserves identity and accounts for rotational symmetries; CanvasPlace accepts any configuration satisfying the constraints rather than requiring agreement with one reference solution. The nested feasible regions and retained feasibility witnesses give a principled difficulty manipulation. Together, the tasks distinguish recovering observed geometry from constructing a valid placement, while exposing the limitations of coarse scene-overlap metrics.

  2. A substantial and appropriately qualified diagnostic study. Language-only adaptation improves prediction while keeping visual features fixed, and oracle-conditioned probes show that angle and scale are more decodable than the base models' autonomous outputs suggest. Blank-image controls, source-page-disjoint splits, edits, and activation interventions provide complementary evidence. The angle-shift study reports Qwen M3 Joint success dropping from 70.24% to 3.33% (Table 4), usefully separating in-distribution fitting from generalization. The authors also acknowledge oracle assistance, unmatched adaptation budgets, and the limits of layerwise causal interpretation.

Critical weaknesses
  1. Evaluation counts and aggregates are internally inconsistent. Section 4.1 and Table 1 specify 200 Tangram scenes per piece count, or 1,400 per model, while Appendix A.3 and Table 8 specify 20 per count. Table 8's GPT-6 Astra Joint values are 100, 100, 100, 90, 85, 60, and 50%; their equal-count average is 83.57%, not Table 1's 87.86%. CanvasPlace has a similar mismatch: Section 4.1 and Appendix B.1 describe 200 evaluated families and 6,000 outputs, whereas Appendix B.2 and Tables 9–10 use 20 families and 600 outputs. The 8/9/12 parse failures yield the reported 96.0/95.5/94.0% parse rates with 200 pooled outputs per tier, not 2,000. Appendix B.1 also refers to both 500 and 50 source families. These differences may arise from unreconciled versions, but they affect effective sample size and the provenance of reported performance. A canonical evaluation manifest and regenerated summaries are necessary to assess the benchmark evidence.

  2. Visibility ambiguity is not quantified for full-pose recovery. Appendix A.1 permits overlapping pieces without collision rejection, yet scoring requires each complete polygon's pose. The known piece count and fixed color identities do not necessarily make heavily occluded geometry recoverable. Consequently, some multi-piece errors may reflect insufficient visual evidence rather than a failure to use available geometry. Appendix F.5 is an important qualification: both base models still have zero target-pose success after oracle isolation, so occlusion cannot explain all of their failures. However, that intervention does not quantify how much the original benchmark's piece-count trends or adaptation differences depend on visibility. A visibility-stratified analysis would delimit the interpretation without requiring a new benchmark.

Questions for authors
  1. Which evaluation manifest and table versions are authoritative for TangramPose and CanvasPlace? Please reconcile the factor-of-ten sample-count differences and the 83.57% versus 87.86% Joint aggregate, identifying any distinct subsets or selection rules and regenerating affected tables from the same records.

  2. After reconciliation, do the principal model comparisons persist when uncertainty is grouped by source page or task family? Please report unique source units separately from augmented scenes, difficulty variants, and model–task outputs. This would establish how much evidence supports the benchmark-level comparisons.

  3. What proportion of targets is heavily occluded at each piece count, and how do the main prediction/adaptation results change on clearly visible targets? Please relate this analysis to the existing oracle-isolation results, which already show that removing clutter alone does not remedy the base models' failure.

8 comments

W1, Q1. Evaluation counts and aggregation

Which records underlie the reported sample sizes, parsing rates and GPT-6 Astra Joint scores, and how were the affected tables regenerated?

Response. We thank the reviewer making these comments. The submitted CanvasPlace bank contained 50 families. We evaluated 20 per tier, giving 200 outputs per tier across ten models. The 8, 9 and 12 parse failures therefore correspond to 96.0%, 95.5% and 94.0% parse success. The manuscript incorrectly reported 500 families and 200 evaluated. For TangramPose, the printed per-count GPT-6 Astra scores averaged to 83.57% because submitted Table 8 gave 50% for seven pieces.

Our data-generation scripts can generate as many TangramPose scenes and CanvasPlace families as needed. We ran a 20-sample-per-group smoke test and a complete 200-sample-per-group evaluation. The repository submitted with the original paper contained the 20-sample evaluation subset, so the submitted tables reported those results; the references to 200 samples per group in the submitted main text were typographical errors. The seven-piece GPT-6 Astra entry in submitted Table 8 was a separate typo: 16/20 scenes passed (80%), making the submitted aggregate 123/140 = 87.86% Joint, consistent with submitted Table 1. The complete 200-sample results are now reported in the revised manuscript. The evaluation procedure and metric definitions remained the same across the two runs.

All ten models share the same 200 seven-piece images. Each model contributes 1,400 TangramPose predictions from 14 held-out source pages. GPT-6’s seven rows in revised Table 8 sum to 1,272 successes (90.86% Joint in Table 1). Each model also contributes 600 CanvasPlace predictions from the same 200 families evaluated at all three tiers. This gives 14,000 TangramPose and 6,000 CanvasPlace predictions across models. The 2 Easy, 3 Medium and 5 Hard parsing failures count as failures among 2,000 outputs per tier (99.90%, 99.85% and 99.75% parse success).

Revised Appendix B.1, lines 1141–1146 (families and tasks), lines 1158–1166 (generation and validation), and lines 1175–1178 (API and parsing); Appendix B.2, lines 1211–1216 (metric denominators). The corrected per-piece-count results are in Appendix A.3, Table 8, lines 810–879.

W2, Q3. Occlusion by piece count and pose recovery on visible targets

How common is heavy occlusion at each piece count, and how do the results on fully visible pieces compare with the target-isolation control?

Response. Thank you very much for providing this comment. Across 60 adaptation-test scenes per piece count, the percentages of target pieces with less than half their area visible were 0%, 0%, 0.56%, 0.83%, 1.00%, 1.39%, and 1.43% for one through seven pieces; 43.81% of seven-piece targets were fully visible. At seven pieces, Qwen M3 recovered 79.37% of all target poses and 92.03% of fully visible poses, while InternVL M3 recovered 62.86% and 74.28%, respectively (Tables 10 and 13). In the separate 200-scene API evaluation, GPT-6 Astra recovered 97.36% of all seven-piece target poses and 98.55% of fully visible poses. Across the same 420 adaptation-test scenes, repacking to remove overlap raised overall Joint from 54.29% to 59.44% for Qwen M3 and from 34.84% to 40.08% for InternVL M3 (Table 12). Repacking also changes piece positions, so this comparison cannot isolate occlusion. In the original paper's oracle-isolation control (Appendix F.5), both Base models still had 0% target-pose success after all other pieces were removed. Thus, limited visibility can contribute to errors, but it does not explain the Base-model failures.

Table 10. Pose success on fully visible targets by piece count (%). Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

Modelk=1k=2k=3k=4k=5k=6k=7
Qwen Base0.000.000.000.000.000.000.00
Qwen M191.67 ± 7.2661.00 ± 12.5348.21 ± 11.2646.88 ± 12.0844.71 ± 9.8347.22 ± 8.5136.23 ± 10.60
Qwen M290.56 ± 1.9287.00 ± 1.0085.13 ± 1.1884.95 ± 1.6289.22 ± 1.7087.90 ± 2.4183.70 ± 0.94
Qwen M393.33 ± 1.6789.67 ± 1.5394.87 ± 0.8993.98 ± 1.3496.86 ± 0.6893.65 ± 1.2492.03 ± 0.83
InternVL Base0.000.000.000.000.000.000.00
InternVL M171.11 ± 13.4749.67 ± 7.0943.08 ± 6.1142.15 ± 3.3138.82 ± 4.8137.90 ± 6.0228.99 ± 2.45
InternVL M271.67 ± 4.4164.67 ± 1.5367.95 ± 6.6370.11 ± 3.9466.08 ± 1.2266.67 ± 1.7963.95 ± 2.26
InternVL M387.78 ± 0.9676.33 ± 2.5280.26 ± 1.1883.01 ± 1.3480.59 ± 4.1277.58 ± 2.4874.28 ± 0.83

Table 11. Scene Joint on standard layouts by piece count (%). Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

Modelk=1k=2k=3k=4k=5k=6k=7
Qwen Base0.000.000.000.000.000.000.00
Qwen M191.67 ± 7.2635.56 ± 19.4610.00 ± 5.005.56 ± 3.852.22 ± 1.921.67 ± 1.670.00 ± 0.00
Qwen M290.56 ± 1.9271.67 ± 2.8955.56 ± 2.5533.33 ± 2.8922.78 ± 3.4712.78 ± 1.9210.00 ± 3.33
Qwen M393.33 ± 1.6773.89 ± 3.4775.00 ± 1.6751.67 ± 1.6746.67 ± 1.6725.00 ± 3.3314.44 ± 4.19
InternVL Base0.000.000.000.000.000.000.00
InternVL M171.11 ± 13.4721.11 ± 3.476.67 ± 0.002.22 ± 0.960.56 ± 0.960.00 ± 0.000.00 ± 0.00
InternVL M271.67 ± 4.4145.00 ± 3.3326.11 ± 3.4719.44 ± 3.478.89 ± 0.960.56 ± 0.962.22 ± 0.96
InternVL M387.78 ± 0.9657.78 ± 3.4742.22 ± 4.1933.33 ± 2.8912.22 ± 1.928.33 ± 2.892.22 ± 2.55

W2, Q3. Occlusion by piece count and pose recovery on visible targets (continued)

Table 12. Pose and Joint success on the same 420 layouts (%). Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

ModelStd. poseRepacked poseStd. JointRepacked Joint
Qwen Base0.040.000.000.00
Qwen M150.32 ± 9.5729.57 ± 4.5720.95 ± 5.2419.05 ± 3.70
Qwen M279.78 ± 0.7081.59 ± 1.6842.38 ± 0.2444.13 ± 1.55
Qwen M386.41 ± 0.5988.67 ± 0.2654.29 ± 0.8659.44 ± 1.59
InternVL Base0.130.000.000.00
InternVL M140.73 ± 5.4824.96 ± 5.4814.52 ± 2.1213.02 ± 2.21
InternVL M261.76 ± 1.1065.29 ± 0.6624.84 ± 0.9624.92 ± 0.77
InternVL M373.71 ± 0.7878.76 ± 0.3034.84 ± 1.3540.08 ± 0.36

For pose success we average the fraction of correct pieces within each scene over 420 scenes (1,680 target pieces in total). Joint gives one score per scene.

Table 13. Pose success on all targets by piece count (%). Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

Modelk=1k=2k=3k=4k=5k=6k=7
Qwen Base0.000.000.000.000.000.280.00
Qwen M191.67 ± 7.2660.83 ± 12.1044.81 ± 11.6842.78 ± 12.0942.44 ± 9.8938.15 ± 10.5231.59 ± 6.69
Qwen M290.56 ± 1.9285.00 ± 1.4481.30 ± 1.1678.06 ± 1.9778.89 ± 1.3573.80 ± 0.9870.87 ± 1.79
Qwen M393.33 ± 1.6786.94 ± 1.7390.74 ± 0.6485.97 ± 0.4887.11 ± 0.1981.39 ± 0.5679.37 ± 1.31
InternVL Base0.000.000.560.000.330.000.00
InternVL M171.11 ± 13.4747.22 ± 5.9140.19 ± 7.2538.47 ± 3.6435.33 ± 5.3629.91 ± 5.7022.86 ± 2.12
InternVL M271.67 ± 4.4164.44 ± 2.1065.74 ± 4.5264.03 ± 2.8459.00 ± 1.2055.37 ± 1.6752.06 ± 2.25
InternVL M387.78 ± 0.9676.39 ± 1.7375.00 ± 1.4776.81 ± 0.8769.89 ± 2.4667.22 ± 1.2162.86 ± 1.49

Compare each column with the fully visible subset in Table 10. Each column covers 60 scenes, with 60 times the piece count target pieces.

Reviewer_yHBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how well VLMs predict precise geometry using two tasks: TangramPose, where models identify pieces and recover their poses, and CanvasPlace, where they place an object subject to geometric constraints. The authors evaluate ten VLMs and compare in-context examples, trained geometric heads, and fine-tuning of language and visual components. They also test whether improvements carry over to unseen rotations, larger numbers of pieces, and a different task. Probing and intervention experiments suggest that visual representations contain geometric information that models do not reliably use in their answers. Fine-tuning improves reconstruction on the training distribution, but these gains do not consistently carry over to the other settings tested.

Critical strengths
  1. The two tasks provide complementary, objectively verifiable evaluations. TangramPose measures recovery of an observed configuration, whereas CanvasPlace measures construction of a feasible configuration and accepts any valid solution. The nested feasible regions in CanvasPlace provide a controlled difficulty intervention while preserving the target, other objects, and instructions. This is more informative than relying exclusively on coarse spatial relations or scene-level overlap.

  2. The diagnostic study goes substantially beyond a leaderboard. Language-only adaptation improves reconstruction while leaving the visual encoder and interface frozen, and the authors investigate this finding using probes, blank-image controls, object-level counterfactuals, and activation interventions across two architectures. The manuscript also appropriately acknowledges important limitations: oracle-assisted probing does not establish autonomous competence, patching does not identify a uniquely necessary circuit, and the head/adaptation comparisons are not compute-matched. These qualifications strengthen the interpretation of the empirical observations.

Critical weaknesses

1.The reported evaluation sizes and results need clarification. The main text reports 200 TangramPose scenes per piece count, while Appendix A.3 reports 20. The per-count GPT-6 Astra results in Table 8 average to 83.57% Joint success, rather than Table 1’s 87.86%. CanvasPlace similarly alternates between 200 and 20 evaluated families, with the reported parsing rates matching the smaller sample size. These discrepancies make it difficult to assess the reliability of the results. The authors should reconcile the counts and explain whether the tables use different evaluation sets.

2.Occlusion may make some target poses ambiguous. Appendix A.1 allows overlapping pieces but evaluates their complete poses, without reporting how much of each piece remains visible. Under heavy occlusion, the image may not uniquely determine the ground-truth pose. The decline in performance with more pieces could therefore partly reflect reduced visibility rather than weaker geometric reasoning. Results on non-overlapping scenes or a breakdown by visibility would help separate these effects.

Questions for authors

1.Which sample counts are correct for TangramPose and CanvasPlace? Please reconcile the main text and appendix, including the GPT-6 Astra Joint score, and provide the underlying success/failure counts.

2.Can occlusion make some poses ambiguous? How does performance change on non-overlapping or clearly visible pieces, especially in larger scenes?

3.Could you report confidence intervals based on independent source pages or families, and check whether the adaptation gains hold across training seeds?

4.Can the transfer evaluation separate geometric accuracy from output-format compliance? Qwen’s results remain uncertain, while InternVL’s gains are confounded by parsing failures, making the extent of geometric transfer unclear.

7 comments

W1, Q1. Evaluation counts and benchmark results: Which sample counts are correct, how do the GPT-6 Astra results reconcile across tables, and what are the underlying counts?

Response. We thank the reviewer providing this comment. The submitted scores used 20 TangramPose scenes per piece count and 20 CanvasPlace families per difficulty tier. The main text incorrectly described this as 200 per group. A separate typo in the seven-piece row of submitted Table 8 gave GPT-6 Astra 50% Joint. It passed 16 of the 20 scenes (80%), which yields 87.86% Joint across all piece counts, as in submitted Table 1.

In the updated evaluation with 200 samples per group, GPT-6 Astra passed 1,272 TangramPose scenes and failed 128 (90.86% Joint). It passed 200 Easy, 200 Medium and 196 Hard CanvasPlace tasks. Across ten models, 1,201 Easy, 884 Medium and 588 Hard placements passed, with 2,000 predictions per tier (Table 1 below). Revised Appendix A, Table 8 gives the TangramPose breakdown by piece count, and Appendix B, Tables 9–10 give the CanvasPlace breakdowns by difficulty and constraint.

Revised manuscript: Section 4.1, lines 224–228 (evaluation sizes); Section 4.2, lines 254–265 and Table 1, lines 271–286 (updated benchmark); Appendix A.3, Table 8, lines 810–879, and Appendix B.2, Tables 9–10, lines 1218–1226 and 1242–1272 (full results).

Table 1. TangramPose and CanvasPlace. Each model receives 1,400 TangramPose scenes (200 per piece count, k = 1–7) and 200 CanvasPlace families at each of three difficulty tiers. Scene IoU and Joint average over scenes. Angle error is in degrees. Scale error, Joint and placement success are percentages. Each Δ is the change in percentage points between successive tiers.

ModelTask 1 TangramPose: Position error ↓Angle error (°) ↓Relative scale error (%) ↓Scene IoU ↑Joint ↑Task 2 CanvasPlace Placement: EasyMediumHardΔ_{E→M} (pp)Δ_{M→H} (pp)
GPT-6 Astra0.0101.700.770.97090.86100%100%98%+0-2
GPT-5.6 Luna0.57272.1868.670.3930.1440%24%12.5%-16-11.5
GPT-4o1.49073.3472.110.2790.0047%21%6%-26-15
Gemini 3.1 Pro1.30366.0456.450.3930.5783.5%72%51.5%-11.5-20.5
Gemini Flash Lite28.18468.5865.060.3640.0773%41%21%-32-20
Claude Sonnet 4.60.81869.7357.760.3812.7986.5%75.5%55%-11-20.5
Qwen3-VL-8B2.92075.1740.860.2090.0025.5%11%3%-14.5-8
Qwen3-VL-30B-A3B1.39974.2031.240.2970.0041%18.5%8%-22.5-10.5
Gemma-4-31B1.26170.3677.150.3610.4363.5%61%34%-2.5-27
Llama-4 Maverick1.59070.8557.320.2720.0740.5%18%5%-22.5-13
Mean3.95564.2152.740.3929.4960.05%44.2%29.4%-15.85-14.8

W2/Q2. Occlusion and pose recovery

How often are pieces heavily occluded as piece count increases? How well do models recover poses when pieces are fully visible, do not overlap, or are shown alone?

W2, Q2 Response. We thank the reviewer providing this comment. Overlap can leave several poses consistent with the visible pixels. We measured the fraction of each piece’s area visible in the original image. Across 60 adaptation-test scenes per piece count, the percentages of pieces with less than half their area visible were 0%, 0%, 0.56%, 0.83%, 1.00%, 1.39%, and 1.43% for one through seven pieces, respectively. In seven-piece scenes, 43.81% of pieces were fully visible.

We compared a geometry head on frozen visual features (M1), language-only LoRA (M2), and adaptation of language, late vision and the cross-modal interface (M3). On those seven-piece scenes, Qwen M3 recovered 79.37% of all target poses and 92.03% of fully visible poses. For InternVL M3 the corresponding rates were 62.86% and 74.28%. On fully visible targets, both architectures retained the ordering M3 > M2 > M1 > Base, with both base models at 0% (Tables 10 and 13). In the separate seven-piece API evaluation, GPT-6 recovered 97.36% of all target poses and 98.55% of fully visible poses, with 85.50% Joint success across whole scenes. Only 19 targets had less than half their area visible, limiting comparisons in that subgroup (Table 3).

Repacking the seven-piece scenes to remove overlap raised whole-scene Joint success from 14.44% to 27.78% for Qwen M3 and from 2.22% to 7.22% for InternVL M3 (Table 2). Joint requires every piece in a scene to pass. Repacking also changes piece positions, so the comparison cannot isolate the effect of occlusion. Both base models remained at 0% Joint on repacked scenes. Even when all other pieces were removed, both had 0% target-pose success (original paper, Appendix F.5). Thus, limited visibility can contribute to errors, while failures persist even for isolated targets. Tables 11 and 12 give whole-scene results by piece count and the full layout comparison.

Table 2. Scene Joint and seven-piece target-pose success (%). Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

ModelStd. JointRepackedStd. k = 7Repacked k = 7All targets k = 7Visible targets k = 7
Qwen Base0.000.000.000.000.000.00
Qwen M120.95 ± 5.2419.05 ± 3.700.00 ± 0.000.00 ± 0.0031.59 ± 6.6936.23 ± 10.60
Qwen M242.38 ± 0.2444.13 ± 1.5510.00 ± 3.338.89 ± 2.5570.87 ± 1.7983.70 ± 0.94
Qwen M354.29 ± 0.8659.44 ± 1.5914.44 ± 4.1927.78 ± 6.7479.37 ± 1.3192.03 ± 0.83
InternVL Base0.000.000.000.000.000.00
InternVL M114.52 ± 2.1213.02 ± 2.210.00 ± 0.000.00 ± 0.0022.86 ± 2.1228.99 ± 2.45
InternVL M224.84 ± 0.9624.92 ± 0.772.22 ± 0.961.67 ± 1.6752.06 ± 2.2563.95 ± 2.26
InternVL M334.84 ± 1.3540.08 ± 0.362.22 ± 2.557.22 ± 0.9662.86 ± 1.4974.28 ± 0.83

Each layout uses the same 420 case IDs, including 60 seven-piece scenes. At k=7 there are 420 targets, including 184 fully visible. Percentages in the last two columns use targets as the denominator.

Table 3. Seven-piece target pose success by visibility (%)

ModelAll (1,400)Fully visible (619)Below 50% visible (19)
GPT-6 Astra97.3698.5594.74
GPT-5.6 Luna0.210.000.00
GPT-4o0.000.000.00
Gemini 3.1 Pro0.790.480.00
Gemini Flash Lite0.500.165.26
Claude Sonnet 4.60.790.000.00
Qwen3-VL-8B0.000.000.00
Qwen3-VL-30B-A3B0.570.320.00
Gemma-4-31B0.360.160.00
Llama-4 Maverick0.210.480.00

Q3. Uncertainty and independent training seeds

Could you report confidence intervals based on independent source pages or families, and check whether the adaptation gains hold across training seeds?

Response. We thank the reviewer for making this comment. We computed pointwise 95% percentile intervals using 5,000 bootstrap draws, resampling whole source pages for TangramPose and whole scene families for CanvasPlace. GPT-6 Astra’s Joint success was 90.86% (95% CI 89.57–92.14%), and its Hard placement success was 98.00% (96.00–99.50%). For adaptation, we retrained M1, M2 and M3 with three independent training seeds on both architectures. Every seed gave M3 > M2 > M1 > Base on overall Joint. Tables 4 and 5 report means and sample standard deviations across these seeds.

Table 4. Joint success on 420 TangramPose scenes (%). Values are the mean ± sample SD across three training seeds.

MethodQwenInternVL
M120.95 ± 5.2414.52 ± 2.12
M242.38 ± 0.2424.84 ± 0.96
M354.29 ± 0.8634.84 ± 1.35

Both Base models score 0% Joint on the same test split.

Table 5. TangramPose performance on standard layouts. Adapted rows give mean ± sample SD over three training seeds. Base was evaluated once.

MethodPosition errorAngle error (°)Scale error (%)Scene IoUJoint (%)
Qwen Base2.92775.6540.210.2100.00
Qwen M10.604 ± 0.0826.32 ± 1.475.39 ± 0.600.557 ± 0.04220.95 ± 5.24
Qwen M20.148 ± 0.00115.53 ± 1.464.04 ± 0.030.833 ± 0.00242.38 ± 0.24
Qwen M30.173 ± 0.0318.08 ± 0.362.80 ± 0.030.867 ± 0.00254.29 ± 0.86
InternVL Base3.14473.4360.870.2350.00
InternVL M10.690 ± 0.0498.22 ± 1.146.61 ± 0.470.518 ± 0.02214.52 ± 2.12
InternVL M20.144 ± 0.00136.78 ± 1.313.78 ± 0.000.768 ± 0.00524.84 ± 0.96
InternVL M30.081 ± 0.00224.88 ± 1.883.31 ± 0.000.815 ± 0.00634.84 ± 1.35

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.