- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for vision-language models that evaluates their ability to reconstruct 3D scenes from 2D images. In the base form, the model is provided with a series of images of a single scene, taken from multiple viewpoints, from which it has to reconstruct the scene as a set of 3D bounding boxes. To further evaluate the spatial understanding the authors introduce a series of question answering tasks to isolate constituent capabilities needed to fully reconstruct the scene (such as camera pose estimation or relative depth prediction). Finally, a navigation task evaluates spatial planning capabilities. The authors provide baseline evaluation on a vast collection of state of the art models, both open and proprietary, and also show a simple fine-tuning baseline.
Critical strengths
- the problem of 3D scene reconstruction from (multiple) 2D views is an important issue. Until recently it was mostly addressed with dedicated method, but is currently in many cases solvable with general purpose MLLM/VLM models with high accuracy. Thus creating VLM specific benchmarks for this task is indeed needed.
- the proposed benchmark (or rather set of benchmarks) is designed to provide a thorough and extensive evaluation of VLM capabilities for 3D scene reconstruction and understanding, from the base image to bounding box prediction task, through question answering to offline navigation scenarios. This allows not only to test if the reconstruction itself can be done by the model, but also evaluates wider understanding of the spatial information extracted from the images.
- the "structure" and "metric" metrics separate the relative layout quality of the reconstruction from physical scale prediction, which is a useful distinction to better understand the VLM predictions.
- provided evaluation is extensive, main results provide a vast collection of baseline methods, while additional experiments and qualitative examples allow to better understand the capabilities of the current models. The results are consistent with overall expectations.
Critical weaknesses
- the main claim of scene reconstruction is largely overstated - what the benchmark actually evaluates is only high-level layout estimation, with only a very coarse bounding box representation as the output, and even object identities are provided to the model as input. There is no object recognition, no surface geometry, texture or other visual appearance evaluation. I would rather describe the task as key object-level 3D layout reconstruction than the standard 3D scene reconstruction, which in the wider embodied AI/robotics community usually means 2D images -> 3D point cloud/mesh/NeRF/GS task. This is also true for the navigation task, which once again is a simplified 2D map navigation scenario. This overstatement needs to be addressed before the paper is accepted.
- at the time of the review, the benchmark data and code is not available - I am not able to ascertain the soundness of the evaluation at the time. At the same time Appendix C mentions a rather implementation-specific details like json file format, which is rather odd for a no-code submission (LLM text writing artefacts?).
- the benchmark doesn't seem to isolate the impact of the multi-view information - the presented tasks, at least according to the qualitative samples in the paper can largely be solved with only a single view. It's not clear if the multi-view information is actually combined by the LLM effectively.
- fine-tuning is limited to only SFT, while spatial tasks often benefit from reasoning capabilities of the model, that require RL-based fine-tuning such as GRPO to train.
- limited number of test cases per category, most notably the offline navigation is only 346 cases, which is quite low for a static benchmark.
- no human performance baseline, or human annotation verification protocol mentioned.
Questions for authors
- What is the number of unique physical scenes used to create the benchmark (not test cases or images, the physical rooms/scenes)?
- Was the full benchmark suite or a subset verified by a human? Can you provide a human baseline for at least the questions answering part as a reference point?
- Can you provide an experiment showing if the multi-view information is genuinely used by the models (e.g. single-view vs multi-view or number of images/performance plot, with the most relevant image as the main single view)
- Can you provide the benchmark data and code for review?
7 comments
Response to the Reviewer Kjro - Part 2
W4. SFT and RL-based fine-tuning
We appreciate this helpful suggestion!
To evaluate RL-based fine-tuning, we extended SFT-50K with Group Sequence Policy Optimization (GSPO) [1], a group-relative RL method related to GRPO, using 2,000 examples (one epoch, 64 queries per batch, eight responses per query). Rewards use binary correctness for MCQs, mean relative accuracy for numerical QA, and for reconstruction, with reward if and zero otherwise.
| Model | Spatial Avg. | General Avg. | GP | SR | Reconstruction | Nav. |
|---|---|---|---|---|---|---|
| Base | 60.98 | 72.09 | 38.44 | 20.03 | 31.00 | 8.10 |
| SFT-50K | 65.13 | 74.58 | 54.22 | 43.37 | 40.67 | 10.40 |
| SFT-50K + GSPO-2K | 67.19 | 75.78 | 55.53 | 46.21 | 44.33 | 12.14 |
Across the same external evaluations, GSPO improves all 10 spatial benchmarks, 7/8 general benchmarks, and all four ReconSpatial task groups over SFT, extending the transfer study with an RL-based baseline. The training setup and results are included in Appendix H.1 (Table 18).
[1] Group Sequence Policy Optimization, arXiv 2025
I thank the authors for providing most of the requested information. The scene diversity clarification, human verification and human baseline, cross-view coverage and cross-view control, as well as GSPO baseline are all important results, supporting the claims. On the negative side, the access to data is pending "institutional approval", therefore I still cannot inspect the data myself. I am leaning towards increasing my score by one, I'll await other reviewers’ responses and make my final decision then. Once again, I thank the authors fot their efforts.
Response to the Reviewer Kjro - Part 1
We apologize for the overly broad task description. We agree that our terminology did not clearly convey the object-level scope. To clarify the scope, ReconSpatial evaluates joint object-level 3D layout reconstruction and offline, open-loop 2D ground-plane navigation. Oriented boxes encode object centers, extents, and orientations. Object identities are supplied by design to decouple object recognition from 3D spatial reasoning, allowing direct evaluation of joint geometric consistency and camera-anchored placement.
ReconSpatial includes 2,574 evaluation instances across 493 source scenes/captures:
Reconstruction spans 71 ScanNet scenes, 98 CA-1M captures, 19 tabletop scenes/sequences, and four nuScenes driving sequences covering 126 outdoor keyframes. Navigation covers 51 ScanNet scenes, with 85.8% of episodes requiring obstacle detours and reference routes of 1.20–11.02 m. Together, these evaluations cover indoor, tabletop, and outdoor environments at different physical scales, alongside varied obstacle-avoidance demands.
Human verification. The full benchmark suite underwent manual quality review: each question was checked against its corresponding ground truth to assess the correctness and consistency of the task and reference answer.
Human reference. On a balanced sample of 400 MCQs across eight atomic tasks, excluding metric-depth estimation because precisely estimating numerical depths in metres from images is difficult for humans, human accuracy reaches 87.75% overall. We compare GP/SR scores with the top two models on the overall leaderboard:
Model values are full-benchmark averages over the corresponding MCQ task types. The absolute accuracy gaps are 4.75–16.17% on GP and 25.56–29.20% on SR in favor of the human reference.
We thank the reviewer for this insightful suggestion!
Cross-view target coverage. To assess whether one view can cover the full target set, we audited all reconstruction instances. At least 81.2% (682/840) require multiple views for complete coverage under the construction criteria, as queried objects are distributed across frames (Figures 5 and 10). Tabletop scenes occupy a compact spatial extent, so one view can cover all targets; other viewpoints provide complementary depth and geometric cues.
Additional geometric information. To isolate the geometric benefit of extra views at fixed target coverage, we compared the best single view against all provided views across indoor, tabletop, and outdoor domains, evaluating only the object subset already visible in the selected single view, with identical queries and reference coordinates. Multi-view minus single-view gains are:
Even for targets already visible in a single view, additional viewpoints improve both Structure and Metric scores across three representative models (+0.02 to +3.38%). Together, these analyses confirm the need for multi-view target coverage in most instances and the geometric benefit of additional viewpoints.
We thank the reviewer for raising this concern. Public release of the benchmark data and evaluation code is undergoing institutional approval. For review, Appendix B.5 provides 12 complete task examples covering all nine atomic QA tasks, reconstruction, and navigation, with visual inputs, exact prompts, and reference answers; Appendix I provides prediction-versus-GT comparisons. Appendix C documents the output interface and scoring rules to support reproducibility ahead of code release. We are committed to releasing the benchmark data and evaluation code upon approval.