acceptodds
Under review as a conference paper at ICLR 2027

DM3SR: Benchmarking Deep Multi-View 3D Spatial Reasoning

Abstract

Vision–language models show strong VQA performance but remain weak at multi-view 3D spatial reasoning. Many existing benchmarks, particularly those built from video-derived frames, provide limited coverage of viewpoint diversity and vertical scene structure, leaving model behavior under broader 3D configurations insufficiently characterized. We introduce DM3SR (Deep Multi-View 3D Spatial Reasoning), a geometry-controlled benchmark spanning three complementary scene families: Contextual, Controlled-Syn and Controlled-Real. DM3SR defines twelve subtasks covering object counting, absolute and relative distance, and lateral and vertical positional reasoning. Evaluating 32 VLMs reveals substantial gaps from human performance, task-dependent benefits from scaling, and inconsistent transfer of spatial adaptation across scenes and axes. Controlled analyses indicate preferences for XY-dominated geometry and continuous viewpoints, alongside difficulty integrating local observations along dimensions with greater scene extent. DM3SR provides a testbed for evaluating cross-view integration and diagnosing how scene geometry and viewing conditions shape spatial reasoning.

Primary area:
datasets and benchmarks
Keywords:
Multi-view, 3D, VLM, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vfgqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DM3SR, a geometry-controlled benchmark for multi-view 3D spatial reasoning. It contains 4,775 question-answer pairs over 384 scenes and 6,475 images, spanning contextual synthetic indoor environments, procedurally generated scenes with greater vertical variation, and controlled real-world captures. Twelve subtasks cover counting, absolute and relative distance, and horizontal and vertical positional reasoning. The authors evaluate 32 VLMs and include a matched human evaluation on a 240-question subset. Diagnostic experiments vary vertical object spread, input-view ordering, and view count. The intended contribution is both an evaluation resource and a means of investigating how scene geometry and viewing conditions affect spatial reasoning.

Critical strengths
  • One aspect I particularly appreciate is that DM3SR allows active control of the 3D object layout and the camera viewpoints, including broader elevation coverage and stronger variation along the Z axis. This makes the benchmark suited for testing spatial reasoning outside the usual XY-dominated indoor distribution.

  • I also like that the authors use this geometric control to probe specific biases in multi-view spatial reasoning, including the vertical scene spread, viewpoint continuity, and the number of input views.

Critical weaknesses
  • Current experiments do not sufficiently isolate the effects of cross-view integration. This is my main concern, and it is closely related to S2. The Full-Single compares full-view accuracy against the average accuracy across individual views. A higher full-view accuracy does not necessarily mean that the model is integrating complementary information from multiple views. It could simply be that one of the views contains enough information to answer the question, and the model benefits from having access to that view. In other words, current experiments do not clearly distinguish between finding the most informative view and actually combining information across different views. I think this distinction is important, especially since cross-view integration is one of the main motivations of the benchmark. The following additional experiments may better support the claim:
  1. Evaluating questions that require multiple views. For example, we can design a question where view A shows an object A and a reference object R, while view B shows an object B and the reference R. The question asks whether A or B is closer to R. Since neither single view alone provides enough information, the model has to combine evidence from both views. This provides stronger evidence of cross-view integration.

  2. Reporting more detailed single-view results, such as best-single and worst-single performance, and the percentage of questions where all individual-view predictions are incorrect but the full-view prediction is correct. Such analysis would help reveal whether the full-view improvement comes primarily from access to a particularly informative view.

  • Some other diagnostic conclusions are also stronger than what the experiments demonstrate. For example, in the vertical-spread experiment, the authors compress the scene along the Z-axis and observe a small improvement in XY positional reasoning. They interpret this as evidence that VLMs prefer XY-dominated geometry. However, changing the vertical distribution of objects may also change other factors, such as occlusion or other visual difficulties. Therefore, it's not sure whether the observed improvement can be attributed to an XY bias. The authors could either provide additional controls to rule out alternative explanations or be more careful in interpreting the results.
Questions for authors

The paper extends an interesting direction in spatial reasoning evaluation. Happy to raise my rating if my concerns can be addressed.

4 comments

Thank you for recognizing the value of controllable layouts and viewpoints, and for proposing a concrete complementary-view experiment, which we ran as suggested.

[W1] Multi-view benefit and integration

To address your concern about finding one informative view, we first examine Full/Single/Blind performance by co-visibility in the original benchmark, then test complementary evidence directly and report the requested single-view statistics.

Co-visibility in the original benchmark

A question counts as co-visible if all of its target objects are visible in one original view. We treat a target as visible when at most 50% of its projected mesh is occluded and its unoccluded area covers at least 0.25% of the image. Rates use the object-only MCQs whose targets can be projected reliably: all 561 Controlled-Syn questions and 349 of 412 Contextual questions (the rest involve targets crossing the camera's near plane).

SplitCo-visible / evaluated MCQsRate (%)
Controlled-Syn68/56112.12
Contextual49/34914.04

To test whether the Full−Single benefit persists beyond co-visible questions, we reanalyze the saved responses for both groups under the same criterion. Mean Single averages the single-view outcomes within each question.

SplitModelGroupNFull (%)Mean Single (%)Blind (%)Full−Single (pp)
Controlled-SynQwen3-VL-32BCo-visible6875.0058.7041.18+16.30
Controlled-SynQwen3-VL-32BNon-co-visible49363.2954.8541.18+8.43
Controlled-SynGLM-4.5VCo-visible6870.5958.6847.06+11.91
Controlled-SynGLM-4.5VNon-co-visible49360.6553.5344.42+7.12
ContextualQwen3-VL-32BCo-visible4967.3559.8055.10+7.55
ContextualQwen3-VL-32BNon-co-visible30065.6758.3353.00+7.34
ContextualGLM-4.5VCo-visible4963.2748.9838.78+14.29
ContextualGLM-4.5VNon-co-visible30054.0048.1143.67+5.89

The gap is larger when one view shows all targets, consistent with an informative view contributing part of the benefit. Full−Single remains positive in the non-co-visible group for both models and both splits (+5.89–8.43 pp). Because this grouping is based on target visibility, it does not by itself establish that each question requires cross-view integration. We therefore also evaluate a separate two-view construction in which neither individual view contains both candidates.

Complementary-evidence test

Following your A–R/B–R proposal, we built 60 new two-view Relative Distance questions from distinct Controlled-Syn scenes. View 1 shows the reference R and candidate A but not B; View 2 shows R and B but not A. Neither view alone shows both candidates, and the question asks which candidate is closer to R in 3D. Visibility in each view was verified with object detection and the 3D scene geometry, and model predictions were never used for selection.

ModelFullMean SingleBlindFull−Single (pp)
Qwen3-VL-32B81.6758.3353.33+23.33
InternVL3.5-38B78.3360.8351.67+17.50
Gemma-4-31B75.0065.8358.33+9.17
GLM-4.5V68.3362.5055.00+5.83

All four models gain from combining the two views (+5.83 to +23.33 pp). Blind accuracy stays near the 50% chance level, so text priors do not solve these questions. Since neither view alone shows both candidates, these gains indicate that the models combine complementary evidence across views. We will release the items and visibility checks.

Best and worst Single

As you requested, we also report Best Single, Worst Single and the rate at which Full is correct while every Single is wrong, on the 1,117 original object-only MCQs with no camera reference.

ModelBest SingleWorst SingleFull correct, all Singles wrong
Qwen3-VL-32B80.8429.102.06
GLM-4.5V79.3224.984.12

Best Single is a ground-truth oracle: an item counts as correct if any singleton response is correct. Both statistics also depend on view count and chance correctness; with four views, independent binary guesses would already give 93.75% Best Single and a 3.125% rate of Full correct with all Singles wrong. These response statistics alone therefore cannot separate an informative view from integration, which is why we rely on the complementary-evidence test above for that distinction.

Together, these results show that the multi-view benefit is not limited to finding one informative view: it persists in questions where no single view shows all targets, and in the direct test where neither view contains both candidates, all four models gain from combining the two views.

[W2] Vertical compression and interpretation

We appreciate your caution about interpreting this intervention. The observed Z decline is consistent with the reduced height separation after compression. As you point out, however, the Relative Distance and XY gains could also reflect changes in occlusion or projected relationships.

Target visibility check

To test the occlusion explanation, we compared the same targets and supplied views before and after compression in the 50 paired scenes. We include the 150 Relative Distance and 100 XY questions with object targets, since camera targets have no mask. Exposure is the unoccluded fraction of a target's silhouette, ignoring image-boundary truncation, averaged over targets and views within each question. Intervals are paired 95% bootstrap intervals over scenes.

TaskQuestionsExposure before (%)Exposure after (%)Change (pp) [95% CI]
Relative Distance15049.2049.28+0.08 [−1.61, +1.83]
Position XY10049.8748.59−1.28 [−3.75, +1.02]

On average, target exposure does not increase after compression, and for XY it slightly decreases while accuracy rises. The gains are therefore not explained by an overall reduction in occlusion. Item-level visibility changes and changes in projected relationships may still contribute, so we present the XY-oriented preference as one possible explanation.

Revision of the interpretation

We will revise Sec. 5.1 accordingly. The sentence “These gains suggest that current VLMs reason more effectively when vertical cues contribute less, revealing a preference for XY-dominated geometry” will become “Performance is sensitive to vertical spread in a task-dependent way; an XY-oriented preference is one possible explanation, alongside item-level visibility changes and changes in projected relationships.” The link to training data will likewise be stated as a hypothesis, and the abstract and conclusion will be aligned with this wording.

I'd like to thank the authors for the response. In particular, I appreciate the new experiment of 60 two-view questions, addressing my main concern about the benefit of multi-view integration. I will consider raising my rating to 3.

For paper revision, I believe a deeper analysis of how VLMs actually integrate information across views, beyond benchmarking protocols and metrics from prior spatial reasoning work, would further strengthen this paper, since, multi-view is one of the focuses of this work.

Thank you for your thoughtful follow-up and for suggesting the complementary-view experiment. We are glad that it addressed your main concern, and we appreciate your constructive feedback throughout the discussion.

We agree that a deeper analysis of how models link evidence across views would strengthen the paper. In the revision, we will incorporate the complementary-view experiment and discuss what it shows about cross-view evidence use. We will also design targeted analyses that vary shared reference objects and view overlap to better characterize the cues models rely on, and continue to explore further analyses in this direction.

Reviewer_XVXKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DM3SR is a benchmark of 4,775 templated questions over 384 scenes: rendered HSSD rooms, Blender scenes with OmniObject3D objects on multi-tier supports, and 48 real tabletop captures. Cameras surround the scene and point at its center, so viewpoints cover a much wider range of elevations than video-derived benchmarks. Tasks cover counting, absolute and relative distance, and XY and Z relative position, with object or camera anchors. The paper evaluates 32 VLMs and runs three interventions: halving vertical spread, reordering views, and varying view count. It concludes that DM3SR needs multi-view input more than prior benchmarks, that spatial fine-tunes do not transfer beyond indoor-like scenes, and that VLMs prefer XY-dominated geometry and continuous views.

Critical strengths
  • The main strength is that the scenes come from full 3D models with a controllable generator. This makes paired interventions possible: change only the vertical spread and recompute answers, or change only the image order. Video-derived benchmarks cannot support such controls. If the generator and scene specifications are released, other groups can run sharper versions of these tests.
  • The evaluation is also wide (32 models, including recent API models with dated identifiers). It reports a useful negative result: several spatial fine-tunes do worse than their base models on the controlled scenes. VG-LLM-8B, for example, drops 11.9 points on CONTROLLED-REAL.
Critical weaknesses
  1. The claim that DM3SR requires cross-view integration is not yet shown. There is no filter against single-view-solvable questions, and the evidence is a Full minus Single gap for two models (Table 11). About 70 percent of the multiple-choice questions refer to "the nth image's viewpoint", and the paper does not say how these are handled when only one image is shown. If that image is missing, single-view accuracy drops by construction. Per-subtask results would show which parts of the benchmark test integration.
  2. Some conclusions are confounded by the protocol. Answer-only prompts do not match how SpaceR, ViLaSR and MindCube-CogMap were trained, and counting scores such as 0.2 for MindCube-RawQA point to format failures. Counting exists only in the CONTROLLED splits, so this may explain part of the transfer gap in Section 4.2.2. The Section 5 effects (0.6 to 2.6 points) are reported without confidence intervals.
Questions for authors
  1. How is the single-view condition run for questions that refer to an image not shown? Please report Full, Single and Blind accuracy per subtask, including Delta on object-only subtasks. A large Delta there would resolve most of my first concern.
  2. What are the geometric constraints in Section 3.3.2? What fraction of questions has all targets visible in one view?
  3. Please give per-subtask results and a split-by-task table with counts.
  4. What are the invalid-response rates per model and task? Does the CONTEXTUAL versus CONTROLLED gap for the spatial models remain when counting is excluded and each model uses its native output format? Which reasoning settings were used for the API models?

Suggestions: add paired bootstrap intervals and a random-permutation control for Section 5; the All-Angles Bench author list is scrambled; the SIBench title has no "true";

6 comments

Thank you for your detailed assessment and for recognizing the value of our controllable generator and broad evaluation. Your concrete requests led to several new analyses, reported below.

[W1, Q1] Camera references and per-subtask Full/Single/Blind

Camera-reference handling

Your estimate is correct: 2,658 of the 3,775 MCQs (70.4%) refer to an image by its index. When that view is withheld, the original identity is kept and the response is scored normally, not automatically marked wrong. An existing camera–camera question illustrates the Single condition:

ConditionImagesPromptOptions (answer)
Original (Full)[image 1], [image 2], [image 3], [image 4]“Which image's viewpoint is closer to the viewpoint of the 3rd image?”A: 1st image; B: 4th image (B)
View 4 only[image 4]“Ablation context: the one supplied image is the original image number 4 in the full multi-view sequence. References to numbered images or views in the question retain their original numbering.<br>Which image's viewpoint is closer to the viewpoint of the 3rd image?”A: 1st image; B: 4th image (B)

The question and options are unchanged, and a prefix states the original number of the supplied image and that numbered references keep their original numbering.

Camera-free versus camera-referenced

To check whether missing references explain the gap, we compare the 1,117 camera-free object-only MCQs with the 2,658 camera-referenced ones on the same responses.

ModelSubsetFullMean SingleBlindFull−Single (pp)
Qwen3-VL-32BCamera-free65.5357.1545.39+8.39
Qwen3-VL-32BCamera-referenced59.3751.0151.58+8.36
GLM-4.5VCamera-free60.7952.5043.60+8.29
GLM-4.5VCamera-referenced58.5848.6144.62+9.97

The gap is similar on both subsets, so withheld referents do not appear to be its main source.

Full/Single/Blind by subtask

F/S/B denote Full/Mean Single/Blind accuracy (%), Δ is Full−Single (pp), and object-only rows are bold. Counting and Absolute Distance are numeric tasks and are not part of this MCQ ablation.

SubtaskNQwen3-VL-32B F/S/BΔppGLM-4.5V F/S/BΔpp
relative_distance/cam-cam38465.36 / 49.44 / 51.56+15.9268.23 / 48.69 / 50.52+19.54
relative_distance/cam-obj38164.57 / 57.17 / 54.86+7.4062.20 / 54.48 / 48.29+7.73
relative_distance/obj-cam38457.29 / 48.49 / 51.56+8.8154.95 / 49.08 / 48.96+5.87
relative_distance/obj-obj37572.27 / 63.47 / 53.07+8.8066.40 / 57.41 / 49.87+8.99
relative_position/xy-cam-cam38442.71 / 48.48 / 48.18-5.7733.85 / 38.65 / 30.21-4.80
relative_position/xy-cam-obj35761.06 / 44.88 / 42.58+16.1863.59 / 44.11 / 38.94+19.47
relative_position/xy-obj-obj37246.77 / 44.27 / 40.05+2.5042.74 / 40.78 / 38.44+1.96
relative_position/z-cam-cam38458.33 / 42.96 / 43.49+15.3761.72 / 44.38 / 38.02+17.34
relative_position/z-cam-obj38466.41 / 65.26 / 68.23+1.1565.89 / 60.61 / 57.03+5.28
relative_position/z-obj-obj37077.57 / 63.68 / 42.97+13.8973.24 / 59.30 / 42.43+13.95

As you suggested, the object-only subtypes are the key test, since no camera reference can be withheld there. All three keep positive gaps for both models, and two are large: Relative Distance +8.80 / +8.99 pp and Z +13.89 / +13.95 pp, while XY is smaller at +2.50 / +1.96 pp (Qwen3-VL-32B / GLM-4.5V). XY camera–camera is the only negative subtype (−5.77 / −4.80 pp). For Qwen3-VL-32B, Full even falls below Blind there, possibly because additional views make it harder to adopt the referenced camera's frame. We will report these anchor-specific results rather than one uniform integration requirement.

[Q2] Geometric constraints and co-visibility

Section 3.3.2 applies these Controlled-Syn constraints to avoid ambiguous answers; questions with ambiguous labels are also excluded.

CheckRequirement
Compared distance differenceAt least max⁡(0.55m,0.12dmax⁡)\max(0.55\mathrm{m}, 0.12d_{\max}), with dmax⁡d_{\max} the larger compared distance
Object-height separationAt least 0.35 m
Camera-related height separationAt least 1 m
XY angular marginAt least 22°
View selectionProjected target size ≥18 pixels, frame margin and coverage

A question counts as co-visible if all of its target objects are visible in one original view. We treat a target as visible when at most 50% of its projected mesh is occluded and its unoccluded area covers at least 0.25% of the image. Rates use the object-only MCQs whose targets can be projected reliably: all 561 Controlled-Syn questions and 349 of 412 Contextual questions (the rest involve targets crossing the camera's near plane).

SplitCo-visible / evaluated MCQsRate (%)
Controlled-Syn68/56112.12
Contextual49/34914.04

Under this criterion, 12–14% of object-only questions have one view in which all targets are jointly exposed.

[Q3] Per-subtask results and split counts

Split-by-subtask counts (C: camera, O: object):

SubtaskControlled-SynControlled-RealContextualTotal
Counting374480422
Abs. Dist.0144434578
Rel. Dist. C–C18748149384
Rel. Dist. C–O18748146381
Rel. Dist. O–C18748149384
Rel. Dist. O–O18748140375
XY C–C18748149384
XY C–O18724146357
XY O–O18748137372
Z C–C18748149384
Z C–O18748149384
Z O–O18748135370
Total224464818834775

Below are the per-subtask scores from the original evaluation on all 4,775 questions, showing the largest model of each Table 3 model family. Absolute Distance uses mean relative accuracy (MRA) and the other subtasks use accuracy (%); each score pools its questions across splits.

ModelCountingAbs. Dist.Rel. Dist. C–CRel. Dist. C–ORel. Dist. O–CRel. Dist. O–O
GPT-5.431.0441.5760.9462.4755.7367.20
Gemini-3.5-Flash41.4752.1367.9762.4754.6972.80
Claude-Sonnet-520.8546.4058.5956.1754.9562.67
Qwen2.5-VL-72B17.0639.7255.9955.1250.7859.73
Qwen3-VL-32B34.3637.5365.1063.7856.7772.53
InternVL3.5-38B22.5137.7056.2559.3251.5667.20
Ovis2.5-9B6.6444.1064.3257.4852.6052.27
MiniCPM-V-4.513.2736.7535.9454.3337.5054.13
GLM-4.5V19.4348.1068.2362.4755.2166.67
Gemma-4-31B38.6346.7659.1160.8954.4367.73
SpaceR18.4836.2157.2955.3846.6157.07
ViLaSR19.9137.9156.2555.6448.9656.53
VG-LLM-8B8.2929.7270.3154.3344.0157.33
MindCube-RawQA0.2440.0351.8249.8748.7052.00
MindCube-CogMap3.3219.3148.7052.4944.2751.47
SpaceQwen2.5-VL-3B6.1623.6550.0040.6850.5250.40
ModelXY C–CXY C–OXY O–OZ C–CZ C–OZ O–O
GPT-5.442.4558.2638.9885.6871.0976.76
Gemini-3.5-Flash39.3268.0732.5393.2384.1179.19
Claude-Sonnet-544.7964.4344.6276.3074.4873.24
Qwen2.5-VL-72B35.1648.7437.6355.7366.9368.92
Qwen3-VL-32B43.2361.3447.0457.5565.8977.03
InternVL3.5-38B41.4144.8243.2846.3561.2068.65
Ovis2.5-9B40.8946.2238.7150.2662.5059.19
MiniCPM-V-4.540.1049.5838.7135.6859.1165.95
GLM-4.5V33.8563.0343.2861.2065.6272.70
Gemma-4-31B41.1552.6643.5571.3582.2975.14
SpaceR46.6150.1441.9440.6264.0668.92
ViLaSR45.5751.8244.8944.5364.5861.08
VG-LLM-8B36.9853.2244.3529.4363.2869.19
MindCube-RawQA51.3046.7844.6238.8048.1841.08
MindCube-CogMap50.2643.1443.2836.9849.2232.97
SpaceQwen2.5-VL-3B23.709.8033.6039.8459.6428.92

[W2, Q4] Native protocols, invalid responses and transfer

Native evaluation protocols

Thank you for raising the possibility of protocol mismatch. Our appendix showed only the default answer-only prompt and the MindCube-RawQA variant, which made it appear that all adapted models used answer-only prompts. The other spatial models ran with their original inference code, connected to DM3SR through a dataset adapter for questions, choices and answer instructions. This keeps ViLaSR's drawing/tool loop and prompts, VG-LLM's geometry encoder, the MindCube-RawQA/CogMap response formats and SpaceQwen's model-card chat interface. SpaceR is the exception: our main run used answer-only instructions, so we tested its official reasoning format below. We will document each model's protocol and prompt in the appendix C.2.

Invalid-response rates

Invalid-response rates (%) for the largest model of each Table 3 model family, listing the 12 with a nonzero rate in any task (task sizes in [Q3]; Overall over all 4,775 questions):

ModelCountingAbs. Dist.Rel. Dist.XYZOverall
Gemini-3.5-Flash7.582.251.518.811.323.79
Claude-Sonnet-520.140.350.000.090.001.84
Qwen3-VL-32B0.000.000.000.000.180.04
Ovis2.5-9B16.110.000.260.000.351.59
MiniCPM-V-4.516.1111.7622.7021.2011.5117.78
GLM-4.5V0.000.170.660.180.880.48
Gemma-4-31B0.240.350.460.630.260.42
SpaceR0.000.001.900.000.180.65
ViLaSR0.005.360.200.000.000.71
VG-LLM-8B0.000.000.130.000.000.04
MindCube-CogMap24.8838.242.891.261.328.36
SpaceQwen2.5-VL-3B0.000.006.0439.710.1811.23

GPT-5.4, Qwen2.5-VL-72B, InternVL3.5-38B and MindCube-RawQA have no invalid responses. Invalid cases are format or extraction failures, such as an option letter inside a numeric answer, option text instead of a letter, or reasoning without a final choice; legal but wrong answers count as errors.

Counting. MindCube-RawQA's near-zero counting score is not a format failure: all 422 outputs are valid numbers, but only one is correct. For CogMap, 24.88% of counting outputs are invalid, yet removing a legal option prefix from tagged numeric answers raises its counting accuracy only from 3.32% to 4.03%.

SpaceR native format

We re-ran SpaceR and its Qwen2.5-VL-7B base with SpaceR's official reasoning format on 480 MCQs (160 per split, 16 per subtype).

ModelAnswer-only (%)Native reasoning (%)Change (pp)
SpaceR52.2954.79+2.50
Qwen2.5-VL-7B base55.2153.54−1.67

Under the native format, SpaceR's gain over its base is +5.63 pp on Contextual and −0.94 pp on Controlled, so the contrast still favors Contextual.

Excluding Counting

Excluding Counting and Absolute Distance, we compare each adapted model's gain over its base on the shared Relative Distance/XY/Z MCQs (1,449 Contextual and 2,326 Controlled), and report the difference Δg\Delta g between the two gains.

Adapted modelContextual gain (pp)Controlled gain (pp)Δg\Delta g (pp)
SpaceR (answer-only)+2.35−0.17+2.52
ViLaSR+4.97−1.59+6.56
VG-LLM-8B+3.66−2.06+5.72

All three contrasts still favor Contextual, and equal task-family weighting preserves their direction, so unequal task coverage alone does not explain the pattern. ViLaSR and VG-LLM already run with their native inference, and SpaceR's native-format check above also favors Contextual, so the contrast persists both without counting and under native output formats.

API reasoning settings

We will document these settings:

ModelReasoning / thinking settingOutput tokens
GPT-5.4Reasoning none64
ClaudeLow effort; thinking disabled256
GeminiMinimal thinking64
Reviewer_abqgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DM3SR evaluates multi-view 3D spatial reasoning using synthetic physical scenes, with answers from geometry. Its CONTEXTUAL, CONTROLLED-SYN, and CONTROLLED-REAL splits contain 384 scenes, 6,475 images, and 4,775 questions across twelve subtasks (refer to Fig. 4–6 and Tab. 2). Such design broadens viewpoint coverage and supports controlled changes to geometry and observations. Evaluation of 32 VLMs examines human–model differences, scaling, and spatial adaptation; subsequent analyses vary vertical spread, view order, and view count. In my reading, the main contribution is the combination of an evaluation resource with an editable diagnostic platform.

Critical strengths
  1. Editable scenes is useful. Answers are derived from geometry (Sec. 3), so the same scenes and questions can be re-evaluated under interventions (Sec. 5): the vertical-spread edit recomputes answers from the modified geometry (Fig. 9, Sec. 5.1), while the view-order control keeps the image set and answer fixed and remaps image references. The physically captured split extends the evaluation beyond rendered images, and the promised scene specifications and generation code would allow controlled studies beyond the fixed benchmark.

  2. The experimental results do show some limitations that aggregate accuracy would hide. Tab. 3 and Fig. 7 show that gains from spatial adaptation vary across scene splits and axes, which supports evaluating spatial specialization outside familiar configurations. I also find the task-wise matched Tiny comparison particularly informative: GPT-5.4 scores 79.2 against the human 83.3 on Relative Position Z, but 41.7 against 83.3 on Relative Position XY. Thus, the human–model gap in this comparison is much larger for horizontal perspective-taking than for vertical ordering.

  3. The viewpoint analysis links benchmark design to a practical evaluation issue. Tab. 1 and Fig. 3 quantify broader directional coverage. Figure 11 then shows order sensitivity with the visual evidence preserved: Relative Distance improves for every evaluated model on both distributions. Figure 12 shows that additional views help different tasks differently, including declining Relative Distance performance on CONTEXTUAL. These observations matter when using spatially separated cameras, where neither a favorable image order nor a benefit from adding views should be assumed. I would prefer to take the broad viewpoint distribution as a useful stress test whose relevance depends on the intended application.

Critical weaknesses

Major

  1. From my perspective, the direction metrics in Table 1 pool all unique images per dataset (p.5 L248); they show that DM3SR's cameras are spread over the sphere, not how much the 2–6 views of a given question disagree or how often all referenced objects are co-visible in one view.

  2. For DM3SR itself, the camera–camera and camera–object templates refer to "the {nth} image" (Table 8), however neither the single-view condition in D.1 nor the view-count protocol in Sec. 5.3 says how such references are represented when that view is withheld, whereas D.2.2 remaps them for reordering. Withholding evidence is legitimate, but the resulting gap is interpretable only if the referent stays fixed, and its size depends on how many items reference a withheld view.

  3. The diagnostic results support behavioral sensitivities more directly than the proposed explanations. Section 5.1 reports gains of 1.71/1.94 percentage points for Relative Distance/XY and a 2.56-point decline for Z (p. 8, lines 409–418). It already explains that compression makes height differences less distinguishable. The edit may also change visibility and image-plane relationships, so it does not isolate a preference learned from training geometry. Similarly, Figure 11 establishes order sensitivity without identifying video-training familiarity as its cause, and the CONTEXTUAL/CONTROLLED contrast in Figure 12 does not isolate scene extent from other differences. The body often presents these explanations cautiously; the abstract and conclusion read more decisively. Keeping the explanations as hypotheses would preserve the useful findings while making their evidential scope clearer.

  4. Table 3 combines comparisons whose populations and procedures need clearer documentation. The Tiny human–model comparison is already matched and useful; the human score is 79.0, versus 54.4 and 50.5 for the two evaluated models. However, the broader statements about all models falling far short of humans (p. 6, lines 309–314) also draw on model scores from the full set. Participant counts, annotations per item, and skip handling are not specified. Random/Majority baselines cover only multiple-choice questions, whereas model Overall scores include numerical tasks. The protocol also mentions model-specific variants and exclusion of incomplete runs without making clear how these affect the retained comparisons. These details matter for the human-gap and model-ranking claims; I am not assuming that the retained models were evaluated on different items.

Minor Issues:

  1. "Substantially higher scene and image diversity per QA pair" (p.6 L284–285): "Avg. images per QA" is the number of input views, not diversity, and by unique images per QA MMSI-Bench (1,990/1,000) exceeds DM3SR (6,475/4,775).

  2. Gemma 4-31B" (p.6 L312), "Gemma-4-31B" (Table 3) and "Gemma 4 31B" (Figs. 10–11) are the same model, unifed the statements.

  3. Ego3D-Bench (ICLR 2026; spatially separated outdoor multi-camera views) and MV-RoboBench (fixed multi-camera robot scenes) bear directly on the claim that existing multi-view benchmarks inherit continuous indoor trajectories (p.1 L046–050; p.3 L144–146) and are not cited.

  4. To my knowledge the MMSI-Bench source list in Tables 2/4 (“ScanNet, Matterport3D”) is incomplete (nuScenes, Waymo, DTU and others are also used), and Table 1 does not say how camera directions were obtained for benchmarks that do not release poses.

Questions for authors

Q1. What a withheld view does to a camera-referenced question.

  • Could you walk through one existing camera–camera or camera–object question (e.g., "Which image's viewpoint is closer to the viewpoint of the {nth} image?") before and after a view is removed, showing the image identities, the reference in the question, the options and the answer? I would like to know which of three cases applies: the item is kept and scored wrong because the referent is unobservable; the references are re-indexed so that the referent changes; or the original identity is kept and the model is asked about an image it cannot see.

Q2. What capability does each subtask isolate? (A hypothesis on which I would welcome the authors' view.)

  • The templates in Table 8 differ in a way that task-level reporting hides. The Relative Position Z questions ("which object's center is the highest or lowest", "is {object} above or below the viewpoint of the {nth} image", "which image's viewpoint is the highest or lowest") are invariant to camera yaw and need no reference direction: gravity gives every view the same "up" and the floor or support surface the same zero, so each view yields a height estimate that can be compared across views without registering the views.
  • The Relative Position XY questions ("standing at the {nth} image's viewpoint and facing the scene center, where is …", "suppose you are standing at {object1} and facing {object2}, where is {object3}") define their own reference direction, and horizontal positions have no shared origin or direction across views, so unless all referenced objects are co-visible in one view, answering requires registering the views (recovering relative camera yaw and position) and then a frame transformation.

Q3. Need to discuss: Is the viewpoint distribution a better distribution or a stress test?

  • CONTEXTUAL places cameras on walls, floor and ceiling in proportion to surface area (p.5 L238–240), so a substantial share of views look up from the floor or down from the ceiling. In real world setting, the views of embodied agents are the deployment distribution rather than a bias; for drone inspection, warehouse camera networks or multi-camera workcells, high-elevation and spatially separated views are in-distribution.

Q4 The generator as the primary asset.

  • Will the release include the intervention tooling (vertical/horizontal spread, occlusion, object count) and the scene specifications needed to regenerate paired sets? Have the authors any evidence that training on DM3SR-style synthetic data transfers to existing benchmarks (e.g., MMSI-Bench or VSI-Bench)?

Q5 Concerns about the quality of label in CONTEXTUAL.

  • Table 3 shows that Human accuracy is lowest on CONTEXTUAL, which may reflect ambiguity of the VLM-generated labels
6 comments

Thank you for your careful review and for recognizing the value of editable scenes for paired interventions. Your questions helped us clarify our protocols and sharpen our interpretation.

[W1] Diversity within each question

We agree that pooled coverage does not show how much the views of a single question differ. We therefore compute two metrics within each question and average them over questions: DD, the mean pairwise angle between the question's viewing directions, and within-question UentU_{\mathrm{ent}}, the normalized entropy of its views on the paper's 36 × 18 azimuth–elevation grid. Pool UentU_{\mathrm{ent}} is the paper's dataset-level metric.

DatasetQADD (°)Within-Q UentU_{\mathrm{ent}}Pool UentU_{\mathrm{ent}}
DM3SR4,77491.390.2040.923
MMSI1,00056.230.1260.598
SPAR4,34545.540.1490.814
MindCube21,15496.620.1810.579
View2Space3,59170.330.1560.677

Under these metrics, DM3SR has the highest within-question and pooled entropy. Its mean pairwise angle is second only to MindCube, whose pooled directional coverage is narrower. Poses come from scene annotations for DM3SR and GT for SPAR; VGGT-Omega [1] estimates are used where GT is unavailable.

Target co-visibility

A question counts as co-visible if all of its target objects are visible in one original view. We treat a target as visible when at most 50% of its projected mesh is occluded and its unoccluded area covers at least 0.25% of the image. Rates use the object-only MCQs whose targets can be projected reliably: all 561 Controlled-Syn questions and 349 of 412 Contextual questions (the rest involve targets crossing the camera's near plane).

SplitCo-visible / evaluated MCQsRate (%)
Controlled-Syn68/56112.12
Contextual49/34914.04

Under this criterion, 12–14% of object-only questions have one view in which all targets are jointly exposed.

[W2, Q1] What changes when views are removed

The third case in your Q1 applies: the original identity is kept, the model is asked about an image it may not see, and its response is scored normally. An existing camera–camera question illustrates both the view-count protocol and the single-view condition:

ConditionImagesPromptOptions (answer)
Original (Full)[image 1], [image 2], [image 3], [image 4]“Which image's viewpoint is closer to the viewpoint of the 3rd image?”A: 1st image; B: 4th image (B)
View count: Views 2 and 3 removed (Sec. 5.3)View 1: [image 1], View 4: [image 4]“Which image's viewpoint is closer to the viewpoint of the View 3?”A: View 1; B: View 4 (B)
Single-view: View 4 only (App. D.1)[image 4]“Ablation context: the one supplied image is the original image number 4 in the full multi-view sequence. References to numbered images or views in the question retain their original numbering.<br>Which image's viewpoint is closer to the viewpoint of the 3rd image?”A: 1st image; B: 4th image (B)

In the view-count protocol, each provided image is preceded by a text label with its original view ID, ordinal references become these stable IDs rather than positions in the shortened list, and the prompt states that they apply even when the referenced view is not provided. In the single-view condition, the question and options are unchanged, and a prefix states the original number of the supplied image and that numbered references keep their original numbering.

As you note, the size of the resulting gap depends on how many items reference a withheld view. To test whether withheld references explain the Full−Single gap, we compare the 1,117 camera-free object-only MCQs with the 2,658 camera-referenced ones on the same Full/Single/Blind responses (Mean Single averages the single-view scores).

ModelSubsetFullMean SingleBlindFull−Single (pp)
Qwen3-VL-32BCamera-free65.5357.1545.39+8.39
Qwen3-VL-32BCamera-referenced59.3751.0151.58+8.36
GLM-4.5VCamera-free60.7952.5043.60+8.29
GLM-4.5VCamera-referenced58.5848.6144.62+9.97

The gap is similar on both subsets, so withheld referents do not appear to be its main source.

[W3] Consistent interpretation

We agree that the abstract and conclusion state these explanations more decisively than the body. We will align their tone with the body, presenting the training-geometry, video-familiarity and scene-extent explanations as hypotheses consistent with the observed sensitivities.

[W4] Matched evaluation

Human eval. Three human annotators independently answered all 240 questions in the Tiny subset. We report the mean of their individual scores. No questions were skipped by any annotator. We will clarify these details in the appendix.

Full benchmark. “Incomplete runs are excluded” refers to models whose evaluation was not completed, not to individual responses; every model reported in the paper was evaluated on all 4,775 questions. The model-specific variants are the native inference protocols of the spatial models MindCube-RawQA/CogMap, ViLaSR, VG-LLM and SpaceQwen, which we keep as in their original work to reproduce each model's reported evaluation setup. All other models, including SpaceR, use the same default answer-only prompt (see our response to Reviewer XVXK (W2,Q4) for a native-format check of SpaceR). We will state each model's protocol explicitly in the appendix. Since the Random/Majority baselines cover only multiple-choice questions, we will also report each model's MCQ-only average in the appendix for a direct comparison.

[Q2] XY and Z requirements

We agree with your analysis of XY: unless the referenced objects are co-visible, the model must relate the views and apply the question's reference frame. For Z, gravity supplies a common “up” and reduces the registration burden, but independently estimated heights need not share a metric scale and zero, since camera elevation, perspective and occlusion affect them. Comparing heights seen in different views therefore still requires linking those observations, though not full XY registration. We will distinguish these demands rather than equate Z with XY.

Our per-subtask reanalysis (see our response to Reviewer XVXK (W1,Q1)) fits this distinction. On object–object questions, where no camera image can be withheld, Full−Single is +13.89 / +13.95 pp for Z but only +2.50 / +1.96 pp for XY (Qwen3-VL-32B / GLM-4.5V). The large Z gap indicates that models can combine height cues observed in different views when a shared vertical reference is available. The small XY gap, together with low XY Full accuracy (46.77% / 42.74%), indicates that additional views provide little benefit when the views must be registered and the question's reference frame applied. Registering the views and applying the reference frame, rather than missing evidence, therefore appear to be the main bottleneck for XY.

[Q3] Why broad elevations matter

We agree that near-horizontal, eye-level views can match the deployment distribution for some embodied agents. As you note, high-elevation and spatially separated views are in-distribution for settings such as drone inspection, warehouse camera networks, and multi-camera workcells. In addition, multi-view systems can incorporate exocentric cameras or observers that do not share the agent's height or orientation, as in Ego-Exo4D and EgoExoLearn [2, 3]. Recent benchmarks such as Ego3D-Bench [4] and MV-RoboBench [5] likewise evaluate spatially separated or fixed multi-camera views. DM3SR is therefore a controllable stress test covering a broader range of viewpoints, rather than a model of how frequently each viewpoint occurs in deployment, and its relevance to a given application depends on that application's viewpoint distribution.

[Q4] Generator release and training transfer

Release. We will release the scene specifications, random seeds, QA generation/validation code, and controls for vertical/horizontal spread, occlusion and object count, so paired sets can be regenerated with recomputed answers.

Transfer. We LoRA-fine-tuned Qwen2.5-VL-3B/7B on 800 DM3SR Relative Distance/XY/Z MCQs per setting, drawn from training scenes only. The three settings use Contextual questions, Controlled-Syn questions, or a Mixed set with 400 of each. We then compared each model with its base on identical evaluation items: 522 MMSI questions covering all six positional-relation categories, and the 1,050 MindCube questions evaluated in the paper. External items were never used for training or checkpoint selection. Scores are accuracy (%), with the change from the same-size base in pp.

BenchmarkModelBaseContextualControlled-SynMixed
MMSI3B29.1232.38 (+3.26)31.42 (+2.30)32.76 (+3.64)
MMSI7B29.8932.18 (+2.30)31.03 (+1.15)32.95 (+3.07)
MindCube3B47.4348.10 (+0.67)48.29 (+0.86)48.76 (+1.33)
MindCube7B36.0039.62 (+3.62)37.14 (+1.14)37.71 (+1.71)

Every fine-tuned model improves on both benchmarks, by 0.67–3.64 pp. Mixed is best on MMSI, whereas Contextual is best for 7B on MindCube. These results indicate positive transfer to existing benchmarks; we will add them to the paper, noting that they come from one training seed.

[Q5] Contextual difficulty and labels

We agree that lower human performance warrants checking label ambiguity. Contextual also includes room-scale relations and some four-choice questions, which can raise difficulty. Qwen3.8-27B generates a broad label (e.g., “seating furniture”) and a specific label (e.g., “white wicker sectional sofa”) from four isolated renders of each object, and QA generation excludes targets with repeated broad classes or non-unique specific labels. A manual inspection of about 100 specific labels found no issues. We will describe these safeguards and the manual check, including its sample size, in the paper.

Minor issues

  • [M1] We agree: “Avg. images per QA” counts input views, not diversity, and by unique images per QA MMSI-Bench (1.99) exceeds DM3SR (1.36). We will replace “diversity per QA” with input-view count and correct the comparison.
  • [M2] We will use one Gemma name throughout the text, tables and figures.
  • [M3] Thank you for these relevant omissions. We will cite Ego3D-Bench and MV-RoboBench [4, 5] and narrow the claim about continuous indoor trajectories.
  • [M4] We will correct MMSI's source list, with full lists in the appendix, state in the main text that directions are estimated with VGGT-Omega [1] when GT poses are unavailable, and distinguish provided, nominal and estimated poses.

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.