- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents DM3SR, a geometry-controlled benchmark for multi-view 3D spatial reasoning. It contains 4,775 question-answer pairs over 384 scenes and 6,475 images, spanning contextual synthetic indoor environments, procedurally generated scenes with greater vertical variation, and controlled real-world captures. Twelve subtasks cover counting, absolute and relative distance, and horizontal and vertical positional reasoning. The authors evaluate 32 VLMs and include a matched human evaluation on a 240-question subset. Diagnostic experiments vary vertical object spread, input-view ordering, and view count. The intended contribution is both an evaluation resource and a means of investigating how scene geometry and viewing conditions affect spatial reasoning.
Critical strengths
-
One aspect I particularly appreciate is that DM3SR allows active control of the 3D object layout and the camera viewpoints, including broader elevation coverage and stronger variation along the Z axis. This makes the benchmark suited for testing spatial reasoning outside the usual XY-dominated indoor distribution.
-
I also like that the authors use this geometric control to probe specific biases in multi-view spatial reasoning, including the vertical scene spread, viewpoint continuity, and the number of input views.
Critical weaknesses
- Current experiments do not sufficiently isolate the effects of cross-view integration. This is my main concern, and it is closely related to S2. The Full-Single compares full-view accuracy against the average accuracy across individual views. A higher full-view accuracy does not necessarily mean that the model is integrating complementary information from multiple views. It could simply be that one of the views contains enough information to answer the question, and the model benefits from having access to that view. In other words, current experiments do not clearly distinguish between finding the most informative view and actually combining information across different views. I think this distinction is important, especially since cross-view integration is one of the main motivations of the benchmark. The following additional experiments may better support the claim:
-
Evaluating questions that require multiple views. For example, we can design a question where view A shows an object A and a reference object R, while view B shows an object B and the reference R. The question asks whether A or B is closer to R. Since neither single view alone provides enough information, the model has to combine evidence from both views. This provides stronger evidence of cross-view integration.
-
Reporting more detailed single-view results, such as best-single and worst-single performance, and the percentage of questions where all individual-view predictions are incorrect but the full-view prediction is correct. Such analysis would help reveal whether the full-view improvement comes primarily from access to a particularly informative view.
- Some other diagnostic conclusions are also stronger than what the experiments demonstrate. For example, in the vertical-spread experiment, the authors compress the scene along the Z-axis and observe a small improvement in XY positional reasoning. They interpret this as evidence that VLMs prefer XY-dominated geometry. However, changing the vertical distribution of objects may also change other factors, such as occlusion or other visual difficulties. Therefore, it's not sure whether the observed improvement can be attributed to an XY bias. The authors could either provide additional controls to rule out alternative explanations or be more careful in interpreting the results.
Questions for authors
The paper extends an interesting direction in spatial reasoning evaluation. Happy to raise my rating if my concerns can be addressed.
4 comments
[W2] Vertical compression and interpretation
We appreciate your caution about interpreting this intervention. The observed Z decline is consistent with the reduced height separation after compression. As you point out, however, the Relative Distance and XY gains could also reflect changes in occlusion or projected relationships.
Target visibility check
To test the occlusion explanation, we compared the same targets and supplied views before and after compression in the 50 paired scenes. We include the 150 Relative Distance and 100 XY questions with object targets, since camera targets have no mask. Exposure is the unoccluded fraction of a target's silhouette, ignoring image-boundary truncation, averaged over targets and views within each question. Intervals are paired 95% bootstrap intervals over scenes.
| Task | Questions | Exposure before (%) | Exposure after (%) | Change (pp) [95% CI] |
|---|---|---|---|---|
| Relative Distance | 150 | 49.20 | 49.28 | +0.08 [−1.61, +1.83] |
| Position XY | 100 | 49.87 | 48.59 | −1.28 [−3.75, +1.02] |
On average, target exposure does not increase after compression, and for XY it slightly decreases while accuracy rises. The gains are therefore not explained by an overall reduction in occlusion. Item-level visibility changes and changes in projected relationships may still contribute, so we present the XY-oriented preference as one possible explanation.
Revision of the interpretation
We will revise Sec. 5.1 accordingly. The sentence “These gains suggest that current VLMs reason more effectively when vertical cues contribute less, revealing a preference for XY-dominated geometry” will become “Performance is sensitive to vertical spread in a task-dependent way; an XY-oriented preference is one possible explanation, alongside item-level visibility changes and changes in projected relationships.” The link to training data will likewise be stated as a hypothesis, and the abstract and conclusion will be aligned with this wording.
I'd like to thank the authors for the response. In particular, I appreciate the new experiment of 60 two-view questions, addressing my main concern about the benefit of multi-view integration. I will consider raising my rating to 3.
For paper revision, I believe a deeper analysis of how VLMs actually integrate information across views, beyond benchmarking protocols and metrics from prior spatial reasoning work, would further strengthen this paper, since, multi-view is one of the focuses of this work.
Thank you for your thoughtful follow-up and for suggesting the complementary-view experiment. We are glad that it addressed your main concern, and we appreciate your constructive feedback throughout the discussion.
We agree that a deeper analysis of how models link evidence across views would strengthen the paper. In the revision, we will incorporate the complementary-view experiment and discuss what it shows about cross-view evidence use. We will also design targeted analyses that vary shared reference objects and view overlap to better characterize the cues models rely on, and continue to explore further analyses in this direction.
Thank you for recognizing the value of controllable layouts and viewpoints, and for proposing a concrete complementary-view experiment, which we ran as suggested.
[W1] Multi-view benefit and integration
To address your concern about finding one informative view, we first examine Full/Single/Blind performance by co-visibility in the original benchmark, then test complementary evidence directly and report the requested single-view statistics.
Co-visibility in the original benchmark
A question counts as co-visible if all of its target objects are visible in one original view. We treat a target as visible when at most 50% of its projected mesh is occluded and its unoccluded area covers at least 0.25% of the image. Rates use the object-only MCQs whose targets can be projected reliably: all 561 Controlled-Syn questions and 349 of 412 Contextual questions (the rest involve targets crossing the camera's near plane).
To test whether the Full−Single benefit persists beyond co-visible questions, we reanalyze the saved responses for both groups under the same criterion. Mean Single averages the single-view outcomes within each question.
The gap is larger when one view shows all targets, consistent with an informative view contributing part of the benefit. Full−Single remains positive in the non-co-visible group for both models and both splits (+5.89–8.43 pp). Because this grouping is based on target visibility, it does not by itself establish that each question requires cross-view integration. We therefore also evaluate a separate two-view construction in which neither individual view contains both candidates.
Complementary-evidence test
Following your A–R/B–R proposal, we built 60 new two-view Relative Distance questions from distinct Controlled-Syn scenes. View 1 shows the reference R and candidate A but not B; View 2 shows R and B but not A. Neither view alone shows both candidates, and the question asks which candidate is closer to R in 3D. Visibility in each view was verified with object detection and the 3D scene geometry, and model predictions were never used for selection.
All four models gain from combining the two views (+5.83 to +23.33 pp). Blind accuracy stays near the 50% chance level, so text priors do not solve these questions. Since neither view alone shows both candidates, these gains indicate that the models combine complementary evidence across views. We will release the items and visibility checks.
Best and worst Single
As you requested, we also report Best Single, Worst Single and the rate at which Full is correct while every Single is wrong, on the 1,117 original object-only MCQs with no camera reference.
Best Single is a ground-truth oracle: an item counts as correct if any singleton response is correct. Both statistics also depend on view count and chance correctness; with four views, independent binary guesses would already give 93.75% Best Single and a 3.125% rate of Full correct with all Singles wrong. These response statistics alone therefore cannot separate an informative view from integration, which is why we rely on the complementary-evidence test above for that distinction.
Together, these results show that the multi-view benefit is not limited to finding one informative view: it persists in questions where no single view shows all targets, and in the direct test where neither view contains both candidates, all four models gain from combining the two views.