acceptodds
Under review as a conference paper at ICLR 2027

BridgeGeom: Diagnosing Fine-Grained Geometric Perception under Structural Aliasing

Abstract

Vision-language models often recognize a bridge as suspension, cable-stayed, or arch, but this does not show that they perceive the geometry that identifies the bridge itself. We study this gap under structural aliasing, where objects share a coarse engineering template and identity rests on weak local geometric cues. We introduce BridgeGeom, a diagnostic benchmark of 971 bridges, 7,555 standardized images, and 10,826 questions, whose core tasks fix the structural type so that only geometric evidence can distinguish instances. Scaling from Qwen3.5-4B to 27B lifts name-image assignment from 19.00% to 32.30%, but same-bridge verification rises only from 70.90% to 75.90% and height comparison from 55.40% to 60.30%. A structural equation over the task measurements yields a dominant scale factor that follows a scaling law with model size, and the geometric blind spot hides inside this factor. We separate it by decomposing the factor into general, image-text, and geometric capability, and measure the geometric capability at 1.057 for Qwen3.5-4B, 1.281 for 27B, and 1.396 for GPT-5.6-sol. Retrieval augmentation lifts the image-text capability but never the geometric capability, and the blind spot is thus contained within the scaling law yet separable from overall capability and shortcut use.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Models, Fine-Grained Geometric Perception, Geometric Blind Spots, Cross-View Verification, Multimodal Diagnostic Benchmark, Retrieval-Augmented Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qwtmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BridgeGeom, a VQA benchmark that centers on structural variations in bridges. The authors introduce 7 tasks to stress-test the visual ability of VLMs with respect to fine-grained geometric understanding in the domain of bridges. The authors evaluate a suite of open and closed-source models, and explore the impact of visual degradations.

Critical strengths

Premise: At a high level, I like the premise of using bridges, which vary in fine-grained structural geometry, as a way to stress-test the visual systems of VLMs. I will add that the authors do need to do more to justify this use case (See limitation 1).

Experiments: I appreciate the authors evaluating both open-source and closed-source models. The results from the visual detail ablation are interesting and unexpected. Quantifying the blind spot in S4.4 was also interesting.

Critical weaknesses

Narrow Scope: The most immediate weakness is that the proposed benchmark is limited to bridges. In and of itself this is not necessarily a weakness, but the authors should do more to motivate why this object/category is particularly useful for studying the shortcut behavior they describe. Currently this motivation is limited to "whose geometry is directly constrained by the surrounding terrain" (L93-94). I do believe bridges are an interesting test-bed, but the authors should do some work to show that their findings translate to other objects/generalize. This can be qualitative and does not need to be exhaustive. Such an analysis would lift this paper from "a bridge benchmark" to using bridges as a means to study an underlying phenomenon in VLMs.

Presentation Issues: There are some presentation issues that make certain regions of the paper hard to read. For example, Table 1 is not well-aligned and floating in between two paragraphs. Similarly, there are many regions with inconsistent whitespace (e.g. L79-83, the three stages in Page 4). I recommend the authors to revisit the wording/spacing/formatting to produce a tighter presentation, as was achieved in Pages 8-9.

Issues with tasks: The framing of Task 6 is ill-defined - without knowing the camera parameters (or another relative measurement in the image) it is impossible to truly predict the height of the bridge from a monocular view. As a result, the model has to rely on either priors about the world/average bridge heights or rely on comparisons with visible objects in the scene and prior knowledge of their heights. This seems to be in conflict with the intended goal of the authors, which is to define the task to identify and mitigate shortcuts.

Questions for authors

Please review the weaknesses. I would be willing to adjust my score with some clarification on the generalization of the findings, and clarification or edits to Task 6. I also recommend working on the presentation issues highlighted for a more easy read.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qwtm

We thank the reviewer for the constructive review, and for appreciating the premise of using bridges, the evaluation of both open and closed models, the visual-detail ablation, and the quantification in §4.4. You indicated that clarification on generalization and on Task 6 could change your assessment; we address both first.

  1. Why bridges, and how the findings transfer. Reviewer baZU raised a related concern about generality; we answer it in full there and summarize it here. By geometric structure we mean the complete spatial form of a particular bridge: the shape, proportion, and arrangement of its members, the distribution of paint and material across them, and the way the bridge fits the terrain it spans. The sentence you quote at L93–94 is the starting point of this definition. Bridges are designed for their sites, and the terrain determines pier heights and span layout, so no two bridges share the same geometric structure, while bridges as a whole fall into only a few shared structural categories. Once the category is fixed, only geometric structure can tell two bridges apart, and yet one bridge can look very different across viewpoints. A single question can therefore pose both difficulties at once. The protocol transfers to other site-specific structures such as dams, pairing two views of one dam against another dam of the same type. Standardized structures built repeatedly provide a useful contrast: two lattice transmission towers of the same design have nearly identical members and differ mainly in their surroundings, so the same protocol would test whether a model relies on the site alone. We will add both examples to the revision and state carefully which conclusions are specific to bridges.

  2. Task 6 and monocular height estimation. A single uncalibrated photograph does not determine scale geometrically, but work on monocular metric depth (e.g., ZoeDepth, Metric3D, UniDepth, and DepthLM, which we cite) shows that models can learn scale priors from scene content such as vehicles, vegetation, and pier spacing. T6 measures whether VLMs have acquired this ability and serves as the absolute-scale counterpart to the relative-height task T7; because it is excluded from the §4.4 fit, it also provides an independent check on the scale trend. Table 2 shows that this ability improves with scale but remains limited. Across the dense Qwen3.5 models, MAE falls from 346.72 m at 4B to 206.94 m at 9B and 174.86 m at 27B, and MRE from 273.99% to 122.02%; Gemma-4 falls from 359.41 m (E2B) to 127.52 m (E4B), and GPT-5.6-sol reaches 75.72 m. As noted in §4.2, the smallest Qwen models do not follow this trend. Even GPT-5.6-sol still has a mean relative error of 54.53%. T7 shows the same pattern: accuracy rises from 55.4% at 4B to 60.3% at 27B and 61.2% for GPT-5.6-sol, still close to the 50% chance level. Physical-scale estimation therefore improves with model size but remains far from reliable. We will state the role of T6 and the monocular-depth context explicitly in §3.2.

  3. Presentation. Thank you. The wrapped placement of Table 1 made §3.2 harder to read than we intended. Taking pp. 8–9 as the standard, we will redesign the placement of Table 1 and standardize the spacing at L79–83, in the three construction stages on p. 4, and elsewhere.

We appreciate your openness to revising your assessment. We hope the definition of geometric structure, the transfer examples, and the role of Task 6 set out above address both points, and we would be grateful for an updated score.

I thank the authors for their response. However, I find the response with regards to generalization slightly lacking. As stated in my review, what is missing is an analysis that these findings generalize beyond this extremely narrow domain of bridges. As a result, and after considering the other reviewers' response, I will maintain my score of a 2.

Reviewer_hh9DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a VQA benchmark that involves answering geometric questions about bridges, with a view to studying the geometric recognition ability of VLMs by identifying a problem where answers are determined by geometry. BridgeGeom contains 7.6K images of bridges with ~11K questions across 8 categories. The authors explore different task formats, including naming bridges from memory, matching bridges to categories, and estimating bridge height. The authors test 10 different models ranging in size, including frontier GPT models, and explore a RAG setting. The findings are that generally models perform poorly across all tasks, with improvements as models get larger, and that even frontier models struggle on some of the subtasks.

Critical strengths
  • The problem introduced seems to be challenging to models, even recent generations of frontier models. The dataset may prove useful to people working in certain very specific domains. The authors test several different tasks.
Critical weaknesses
  • The domain of the dataset seems excessively narrow and not convincingly connected to geometry. While there are some motivations for wanting to be able to recognize bridges, this seems like a fairly niche contribution that does not merit a full paper. The geometric reasoning highlighted in the introduction is interesting but underexplored. 4.4 is a bit hard to understand but my reading of it is that it gives predictions about how models will do at different tasks but does not really address the question of what kind of geometric reasoning models are missing which would allow them to tackle this task. It seems to me like this paper proposes a challenging benchmark but the motivation for it (either practical or analysis-level) is weak.

  • Parts of the writing do not make sense and show signs of LLM generation, e.g. L107 (The blind spot is thus contained within the scaling law yet separable from overall capability and shortcut use --> what scaling law is this referring to, this is the first time the scaling law has come up in the intro). In general the content about the scaling law (which seems core to the contribution) is very hard to understand. The references have issues; a lot of the author first names are wrong, for example, DepthLM (arXiv 2509.25413). The paper cites “Zhen Cai et al.” The real first author is Zhipeng Cai; Downscaling Intelligence (arXiv 2511.17487). The paper cites “Mark Endo and Stephanie Yeung-Levy.” The second author is actually Serena Yeung-Levy; The Last Visible Pixel (arXiv 2606.07861). The paper cites “Lei Li et al.” The real first author is Lujun Li.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer hh9D

Thank you for the specific comments, in particular for identifying the errors in our references and for asking what concrete failure the benchmark reveals.

  1. Scope, and what kind of failure the benchmark reveals. Reviewer baZU raised the same concern about scope, and our response there gives the full definition of geometric structure, the rationale for bridges, and the transferable protocol. Here we address where the model fails. The Gokteik Viaduct in Myanmar gives a checkable example. Shown each view alone, Qwen3.5-4B correctly labels both as truss bridges (T2-RAW-1354-2, T2-RAW-1354-3). Shown the same two photographs together, it answers "different bridges" (T5-POS-RAW-1354-2). The steel trestle towers, the deck truss, and the gorge are clearly visible in both views, yet in its reasoning the model describes the trestle towers in one view as concrete piers. Qwen3.5-27B, GPT-5.5, and GPT-5.6-sol also answer "different" on this item. The model recognizes the structural category but misreads the members of this particular bridge, and so fails to recognize it across changes in viewpoint and imaging conditions; the same failure appears in frontier models. Reusing the same photographs across tasks is what allows the benchmark to localize this failure.

  2. The scaling law at L107 and the clarity of §4.4. The scaling law at L107 is the saturating power law G(M) = A − B·M^(−κ) in Eq. (4) of §4.4. It takes the form of the parameter term in the loss law of Hoffmann et al. (2022), a power law approaching an asymptote, with the sign reversed because G increases with scale rather than decreasing. Kaplan et al. (2020) first reported power-law scaling of loss with model size. We cite both in §4.4 but not at the first mention in the introduction, and we did not say there what is being fitted. We will add both. Section 4.4 does three things: it uses d′ to separate a model's ability to tell bridges apart from its tendency to answer "same"; it measures the contribution of regional commonalities as C, the difference between cross-country and same-country negatives; and it fits how same-country discrimination G grows with model size. The revised section will open with this summary.

  3. References and writing. Thank you for identifying these errors. The correct names are Zhipeng Cai (DepthLM), Serena Yeung-Levy (Downscaling Intelligence), and Lujun Li (The Last Visible Pixel). Tracing our workflow, we found that the errors arose in the final step before submission, when GLM converted the reference list into a separate BibTeX file to remove duplicates and uncited entries and rewrote author first names in the process. Before that step, we had opened each DOI or arXiv link and checked the authors and the correspondence with our text. All three errors preserve the correct surname and first initial, and the arXiv IDs and DOIs in these entries are correct, which is consistent with an error in expanding names during conversion. We did not re-proofread after the conversion, and that oversight is ours. [We have now verified every entry against its original record.] This conversion falls within the uses described in our AI usage disclosure at submission (writing assistance and citation handling). In the revision we will expand the disclosure to list each step in which GLM was involved and the manual check that followed it. The pre-conversion bibliography and its version history are available to the AC on request.

On writing style, the paper discloses that GLM was used for polishing and for compressing the text to meet the page limit. The authors read the result and judged it fluent, but the compression removed definitions that readers need at first mention, as at L107. We did not consider the first-time reader enough. In revising, we will reread the paper from the reader's side and make sure every concept is defined and cited where it first appears.

We are grateful for the careful check of our references. We hope the Gokteik example and the clarified §4.4 make the contribution of the benchmark more concrete, and that you might take this into account in your final assessment.

Reviewer_baZUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces BridgeGeom, a benchmark of 7,555 images of 971 bridges, to evaluate the geometric perception of VLMs. The tasks cover name recall, category recognition, name-image matching, same-bridge selection and verification, and height estimation. By holding the structural type unchanged, the benchmark aims to disentangle geometry from other information. The paper evaluates a wide range of VLMs, including retrieval-augmented approaches, showing the perception deficit of existing models.

Critical strengths
  1. The paper researches an important question about the object-level geometry perception ability of VLMs, and contributes a benchmark that probes this question.
  2. The evaluation is extensive, covering several model families and scales.
Critical weaknesses
  1. A great part of the experimental design does not match the stated motivation. The benchmark aims to isolate geometric perception, yet most of its tasks do not test it. T1 and T3 measure entity knowledge, T2 measures category recognition, and T6 and T7 ask for heights from photographs without any scale reference. Even T4 and T5 can be solved from the surrounding environment and background rather than the structure itself. The paper's data also shows increased G from retrieval augmentation.
  2. The title and conclusion speak of general geometric perception, but the evidence only comes from a single narrow domain of bridges.
  3. All results come from single runs, and no uncertainty is quantified.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer baZU

We thank the reviewer for the careful reading and for recognizing the importance of the question and the breadth of our evaluation. We respond to each point below.

  1. Task design and the role of environment and background. Whether T4/T5 can be solved from background depends on what counts as a bridge's structure. By geometric structure we mean the complete spatial form of a particular bridge: the shape, proportion, and arrangement of its members, the distribution of paint and material across them, and how the bridge fits the terrain it spans. Paint follows the members, material sets their form, and terrain sets pier heights and spans (L93), so no two bridges share this form, and it can only be read from the bridge itself. A shortcut in the sense of Geirhos et al. (2020) is instead regional commonality, such as architectural style or vegetation, which narrows the region but cannot single out a bridge. G is computed on same-country negatives, where these commonalities are shared; the extra discrimination on cross-country negatives is C. Removing color leaves G unchanged (1.057 → 1.062) while C falls from 0.552 to 0.273; reducing resolution lowers G to 0.876. Models also fail with the full structure visible: given a hand-tinted postcard and a recent photograph of the Gokteik Viaduct, Qwen3.5-4B labels both as truss bridges yet answers "different", describing the steel trestles as concrete piers; 27B, GPT-5.5, and GPT-5.6-sol also answer "different". T1–T3 and T6/T7 are reference tasks for T4/T5, which fix the category; in 339 T4 questions all candidates share the query's category. For T6, see our response to Reviewer qwtm.

  2. Retrieval augmentation increases G. You are right that four of the five retrieval configurations raise G. Section 4.4 tests a further question: whether the gain on T5 carries over to T4 as the discrimination level predicts. Table 3 shows that every configuration that raises G also widens the T4 prediction gap. EchoSight raises G from 1.057 to 1.518 and T4 accuracy from 77.7% to 80.6%, but its T5 discrimination predicts 87.7%, so the gap widens from 3.2 to 7.1 points; for UniRAG it reaches 19.2 points. MoRE, the only configuration that narrows the gap, lowers G to 0.894. No configuration raises G while preserving closed-book cross-task consistency. The abstract's phrase "never the geometric capability" compressed this finding, and we will replace it with the statement above.

  3. Evidence from a single domain. Bridges fit our definition fully. Each is designed for its site, so their geometric structures differ, yet they share a small number of engineering categories, and one bridge can look very different across viewpoints. A single question can therefore require a model to recognize the same bridge from a new view while rejecting another bridge of the same category. The protocol transfers to other site-specific structures such as dams: positives are two views of one dam, negatives another dam of the same type (e.g., an arch dam), with negatives split by region and degradation controls applied. Standardized structures that are built repeatedly provide a contrast. Two lattice transmission towers of the same design have nearly identical members and differ mainly in their surroundings, so the same protocol would test whether a model relies on the site alone. We will review the statements in the abstract and conclusion that extend beyond bridges, and add a scope paragraph with both examples.

  4. Single runs and uncertainty. On single runs: all models use greedy decoding, so generation involves no random sampling. Both GPT models were also evaluated twice, first on a 10% pilot subset and then on the full benchmark; Table 2 reports the full runs.

On uncertainty: we tested whether the conclusions would hold on a different sample of questions. We resampled the T5 questions with replacement to form a new set of the same size, scored both models on that same set, recorded the difference, repeated this 10,000 times, and took the middle 95% of the differences. Because a bridge appears in several questions, questions that share a bridge were drawn together as one group (307 groups), so that one bridge is not counted as several independent pieces of evidence. Qwen3.5-27B exceeds 4B by 5.0 points in T5 accuracy (95% interval 2.6 to 7.9) and by 0.22 in same-country discrimination G (0.04 to 0.52). Both lower bounds are above zero, so the gain from scale does not depend on which questions happened to be sampled. The released file data/items_T5.csv.gz contains the bridge IDs, gold answer, and each model's response for every question, and data/bridges.csv gives each bridge's country, so the procedure can be reproduced directly. We will add these intervals to the tables.

We hope the definition of geometric structure, the G/C analysis, and the new intervals resolve the soundness concerns you raised; if so, we would appreciate a reconsideration of your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.