- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BridgeGeom, a VQA benchmark that centers on structural variations in bridges. The authors introduce 7 tasks to stress-test the visual ability of VLMs with respect to fine-grained geometric understanding in the domain of bridges. The authors evaluate a suite of open and closed-source models, and explore the impact of visual degradations.
Critical strengths
Premise: At a high level, I like the premise of using bridges, which vary in fine-grained structural geometry, as a way to stress-test the visual systems of VLMs. I will add that the authors do need to do more to justify this use case (See limitation 1).
Experiments: I appreciate the authors evaluating both open-source and closed-source models. The results from the visual detail ablation are interesting and unexpected. Quantifying the blind spot in S4.4 was also interesting.
Critical weaknesses
Narrow Scope: The most immediate weakness is that the proposed benchmark is limited to bridges. In and of itself this is not necessarily a weakness, but the authors should do more to motivate why this object/category is particularly useful for studying the shortcut behavior they describe. Currently this motivation is limited to "whose geometry is directly constrained by the surrounding terrain" (L93-94). I do believe bridges are an interesting test-bed, but the authors should do some work to show that their findings translate to other objects/generalize. This can be qualitative and does not need to be exhaustive. Such an analysis would lift this paper from "a bridge benchmark" to using bridges as a means to study an underlying phenomenon in VLMs.
Presentation Issues: There are some presentation issues that make certain regions of the paper hard to read. For example, Table 1 is not well-aligned and floating in between two paragraphs. Similarly, there are many regions with inconsistent whitespace (e.g. L79-83, the three stages in Page 4). I recommend the authors to revisit the wording/spacing/formatting to produce a tighter presentation, as was achieved in Pages 8-9.
Issues with tasks: The framing of Task 6 is ill-defined - without knowing the camera parameters (or another relative measurement in the image) it is impossible to truly predict the height of the bridge from a monocular view. As a result, the model has to rely on either priors about the world/average bridge heights or rely on comparisons with visible objects in the scene and prior knowledge of their heights. This seems to be in conflict with the intended goal of the authors, which is to define the task to identify and mitigate shortcuts.
Questions for authors
Please review the weaknesses. I would be willing to adjust my score with some clarification on the generalization of the findings, and clarification or edits to Task 6. I also recommend working on the presentation issues highlighted for a more easy read.
2 comments
I thank the authors for their response. However, I find the response with regards to generalization slightly lacking. As stated in my review, what is missing is an analysis that these findings generalize beyond this extremely narrow domain of bridges. As a result, and after considering the other reviewers' response, I will maintain my score of a 2.
Response to Reviewer qwtm
We thank the reviewer for the constructive review, and for appreciating the premise of using bridges, the evaluation of both open and closed models, the visual-detail ablation, and the quantification in §4.4. You indicated that clarification on generalization and on Task 6 could change your assessment; we address both first.
Why bridges, and how the findings transfer. Reviewer baZU raised a related concern about generality; we answer it in full there and summarize it here. By geometric structure we mean the complete spatial form of a particular bridge: the shape, proportion, and arrangement of its members, the distribution of paint and material across them, and the way the bridge fits the terrain it spans. The sentence you quote at L93–94 is the starting point of this definition. Bridges are designed for their sites, and the terrain determines pier heights and span layout, so no two bridges share the same geometric structure, while bridges as a whole fall into only a few shared structural categories. Once the category is fixed, only geometric structure can tell two bridges apart, and yet one bridge can look very different across viewpoints. A single question can therefore pose both difficulties at once. The protocol transfers to other site-specific structures such as dams, pairing two views of one dam against another dam of the same type. Standardized structures built repeatedly provide a useful contrast: two lattice transmission towers of the same design have nearly identical members and differ mainly in their surroundings, so the same protocol would test whether a model relies on the site alone. We will add both examples to the revision and state carefully which conclusions are specific to bridges.
Task 6 and monocular height estimation. A single uncalibrated photograph does not determine scale geometrically, but work on monocular metric depth (e.g., ZoeDepth, Metric3D, UniDepth, and DepthLM, which we cite) shows that models can learn scale priors from scene content such as vehicles, vegetation, and pier spacing. T6 measures whether VLMs have acquired this ability and serves as the absolute-scale counterpart to the relative-height task T7; because it is excluded from the §4.4 fit, it also provides an independent check on the scale trend. Table 2 shows that this ability improves with scale but remains limited. Across the dense Qwen3.5 models, MAE falls from 346.72 m at 4B to 206.94 m at 9B and 174.86 m at 27B, and MRE from 273.99% to 122.02%; Gemma-4 falls from 359.41 m (E2B) to 127.52 m (E4B), and GPT-5.6-sol reaches 75.72 m. As noted in §4.2, the smallest Qwen models do not follow this trend. Even GPT-5.6-sol still has a mean relative error of 54.53%. T7 shows the same pattern: accuracy rises from 55.4% at 4B to 60.3% at 27B and 61.2% for GPT-5.6-sol, still close to the 50% chance level. Physical-scale estimation therefore improves with model size but remains far from reliable. We will state the role of T6 and the monocular-depth context explicitly in §3.2.
Presentation. Thank you. The wrapped placement of Table 1 made §3.2 harder to read than we intended. Taking pp. 8–9 as the standard, we will redesign the placement of Table 1 and standardize the spacing at L79–83, in the three construction stages on p. 4, and elsewhere.
We appreciate your openness to revising your assessment. We hope the definition of geometric structure, the transfer examples, and the role of Task 6 set out above address both points, and we would be grateful for an updated score.