acceptodds
Under review as a conference paper at ICLR 2027

SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation

Abstract

Multimodal Large Language Models (MLLMs) have made rapid progress in spatial intelligence, yet existing spatial reasoning benchmarks largely assume pristine visual inputs and overlook the degradations that commonly occur in real-world deployment, such as motion blur, low light, adverse weather, lens distortion, and compression artifacts. This raises a fundamental question: how robust is the spatial intelligence of current MLLMs when visual observations are imperfect? To answer this question, we introduce SpaceDG, the first large-scale dataset for degradation-aware spatial understanding. It is constructed with a physically grounded degradation synthesis engine that embeds degradation formation process into 3D Gaussian Splatting (3DGS) rendering, enabling realistic simulation of nine degradation types. The resulting dataset contains approximately 1M QA pairs from nearly 1,000 indoor scenes. We further introduce SpaceDG-Bench, an human-verified benchmark with 1,102 questions spanning 11 reasoning categories and 9 visual degradation types, yielding over 10K VQA instances. Evaluating 25 open- and closed-source MLLMs reveals that visual degradations consistently and substantially impair spatial reasoning, exposing a critical robustness gap. Finally, we show that finetuning on SpaceDG markedly improves degradation robustness and can even surpass human performance under degraded conditions without any performance drop on clean images, highlighting the promise of degradation-aware training for robust spatial intelligence.

Primary area:
datasets and benchmarks
Keywords:
Spatial Intelligence, Multimodal Large Language Models, Visual Degradation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M5RbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpaceDG, a 3DGS-based dataset and benchmark for evaluating MLLM spatial reasoning under nine realistic visual degradations. Experiments on 25 models reveal significant robustness gaps, while degradation-aware fine-tuning substantially improves performance on both clean and degraded images.

Critical strengths
  1. The paper addresses an important yet underexplored problem: the robustness of MLLM spatial reasoning under realistic visual degradations, bridging the gap between existing clean-image benchmarks and real-world deployment scenarios.
  2. The proposed 3DGS-based degradation synthesis pipeline generates diverse visual degradations while preserving geometric consistency and spatial ground truth, enabling large-scale construction of reliable spatial QA data.
  3. The authors evaluate 25 MLLMs across nine degradation types and 11 spatial reasoning tasks, revealing substantial robustness gaps and identifying fine-grained object-centric reasoning as particularly vulnerable to visual degradations.
  4. Supervised fine-tuning on SpaceDG substantially improves spatial reasoning performance under both clean and degraded conditions, demonstrating the practical value of the constructed dataset.
Critical weaknesses
  1. Although the paper reports substantial gains from SpaceDG-SFT, the clean-only SFT baseline already achieves comparable performance. Specifically, degradation-aware SFT improves average degraded performance by only 1.3 percentage points (64.8% → 66.1%), raising questions about the unique contribution of degradation augmentation.
  2. The dataset is primarily constructed from indoor ScanNet++ scenes with synthetically generated degradations. The lack of evaluations on real-world degraded images leaves uncertainty about how well the benchmark and training improvements transfer to practical deployment scenarios.
  3. Insufficient experimental evidence for some conclusions. The finding that degradation-guided CoT harms spatial reasoning is supported by experiments on only one model, making it difficult to generalize this conclusion to other MLLMs or reasoning strategies.
Questions for authors
  1. Limited improvement attributable to degradation-aware training According to Table 6, degradation-aware SFT only improves performance by 1.3 percentage points over clean-only SFT (64.8 vs. 66.1) on degraded images. Could the authors further clarify the unique contribution of degradation augmentation and provide additional experiments to demonstrate whether this improvement is statistically significant?
  2. Limited real-world generalizability Since SpaceDG is primarily constructed from indoor scenes with synthetically generated degradations, how well do the proposed benchmark and degradation-aware training generalize to real-world degraded images? Could the authors provide additional evaluations on real-world degradation datasets or outdoor environments to validate the practical applicability of their approach?
  3. Insufficient evidence for the CoT-related conclusion The conclusion that degradation-guided CoT negatively affects spatial reasoning is based on experiments with only Gemini-3.1-Flash-Lite. Could the authors extend this evaluation to additional MLLMs, particularly stronger reasoning models, and investigate whether alternative degradation-aware prompting strategies could improve performance rather than impair it?
6 comments
AuthorsOpenReview ↗

Responses to Reviewer M5Rb's W1&Q1

Dear reviewer M5Rb, we sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. In the following, we provide a detailed response to address your concerns.

W1&Q1. Incremental gain from degradation-aware SFT

Thank you for raising this concern. To examine the benefits of degradation-aware SFT, we extend SFT into a second epoch. As shown in the table, clean-only SFT and SpaceDG-SFT achieve average scores of 67.3 and 69.3, respectively, across the nine SpaceDG-Bench degradations. Relative to epoch 1, SpaceDG-SFT gains 3.2 points, compared with 2.5 points for clean-only SFT, widening its advantage from 1.3 to 2.0 points. This widening suggests that degradation-aware supervision can deliver increasing benefits with continued training. It further motivates the conclusion that, compared with clean-only SFT, degradation-aware SFT can use additional training exposure and computation more efficiently to improve spatial reasoning robustness.

EpochClean-only SFTSpaceDG-SFTDelta
164.866.1+1.3
267.369.3+2.0

We also conduct paired bootstrap analyses with 10,000 replicates at both the scene and question levels using 1,102 questions from 320 scenes, keeping both models' predictions and all nine degradation variants together within each sampled unit. The second-epoch gain is +1.97 points before rounding, corresponding to +2.0 in the table above. Both 95% percentile confidence intervals exclude zero, supporting a positive average advantage of SpaceDG-SFT on degraded images.

Resampling unitDelta95% CI
Scene+1.97[+1.04, +2.92]
Question+1.97[+1.06, +2.89]
AuthorsOpenReview ↗

Responses to Reviewer M5Rb's W2&Q2

W2&Q2. Generalization to degraded images from real-world outdoor scenes

Thank you for highlighting the importance of evaluating real-world outdoor scenes under visual degradation. We evaluate both models on a stratified sample of 500 questions from Ego3D-Bench [1], covering real-world driving scenes from Waymo, nuScenes, and Argoverse. Some of the original images already exhibit naturally occurring degradation, including low light, motion blur, sun glare with clipped highlights, and localized blur in rainy conditions. We evaluate the same questions using the original images and versions with each of the nine SpaceDG degradations applied separately. We report multiple-choice accuracy and average the degraded results across the nine conditions.

ModelInput ImagesDegradation TypesPerformance (%) ↑
Qwen3-VL-8BOriginalLow light, motion blur, sun glare with clipped highlights, and localized blur in rain45.97
Qwen3-VL-8BDegradedNine SpaceDG types42.42
SpaceDG-SFT-Qwen3-VL-8BOriginalLow light, motion blur, sun glare with clipped highlights, and localized blur in rain57.02
SpaceDG-SFT-Qwen3-VL-8BDegradedNine SpaceDG types56.16

As shown in the table, SpaceDG-SFT improves accuracy under the added degradations by 13.74 percentage points over the base model and retains nearly all of its accuracy on the original images. Notably, SpaceDG-SFT is trained only on degraded indoor scenes from ScanNet++, so these results provide evidence that its gains transfer to outdoor driving scenes. In addition, its accuracy decreases by only 0.86 percentage points from original to degraded conditions, compared with 3.55 percentage points for the base model. These results demonstrate that SpaceDG-SFT transfers both its spatial reasoning capability and degradation robustness from indoor training scenes to a substantially different outdoor domain.

[1] Mohsen Gholami et al. Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes. ICLR 2026.

AuthorsOpenReview ↗

Responses to Reviewer M5Rb's W3&Q3

W3&Q3. Degradation-guided CoT with additional MLLMs

ModelMethodPerformance (%) ↑
GPT-6-Astraw/o CoT56.18
GPT-6-Astrawith CoT54.71
Qwen3.8-Flash-Nextw/o CoT54.09
Qwen3.8-Flash-Nextwith CoT53.32

We extend the two-stage CoT method to GPT-6-Astra and Qwen3.8-Flash-Next on SpaceDG-Bench-900, which comprises 100 questions evaluated under each of nine degradations. For each model, we hold the images, questions, and reasoning-effort setting fixed. The CoT prompt asks for a degradation label and a brief spatial explanation before the final answer, as described in Section 5.1. As shown in the table, the overall changes are −1.47 and −0.77 percentage points, respectively, providing further evidence that degradation-guided CoT can impair model performance under visual degradation.

As illustrated in Figures 11–19, our analysis suggests that visual degradation can lead models to misinterpret image content. CoT reasoning then proceeds from incorrect premises, propagating these perceptual errors and ultimately producing incorrect conclusions.

Reviewer_SWNGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpaceDG, a dataset and benchmark for evaluating the spatial reasoning capabilities of MLLM under visual degradation. The authors construct 3D scene representations, derive spatial questions and answers from camera poses and object annotations, and synthesize 9 degradation types using rendering-based methods.

The main contribution is a controlled evaluation resource, accompanied by an empirical study of current MLLMs and the potential benefits of training on the proposed data, providing a useful resource for evaluating and improving MLLM robustness.

Critical strengths
  1. The paper addresses a meaningful problem. Spatial reasoning under visual degradation is relevant to real-world deployment. The dataset covers diverse spatial tasks and degradation types.

  2. Clear and accessible presentation. The paper is well organized and easy to follow. The dataset construction and evaluation protocol are clearly described, and the figures and examples help understand the benchmark.

  3. Comprehensive experimental evaluation. The paper evaluates 25 models across general-purpose, spatial, and robotic model families. The experiments provide an open platform to evaluate the spatial reasoning ability of MLLM in the context of visual degradation and demonstrate the dataset’s potential for spatial training and evaluation.

Critical weaknesses
  1. The attribution of the substantial SFT gains to physically grounded degradations is not sufficiently supported. Table 6 shows that most of these gains are already achieved through clean-SFT, while degradation augmentation adds only 0.1 points on clean inputs and 1.3 points on degraded inputs.

  2. Training details are incomplete. The paper reports the training epochs, batch size, and hardware, but omits important settings such as the learning rate, optimizer, schedule, and module-freezing configuration. These omissions limit reproducibility and assessment of the training.

  3. Lack of comparison with other similar benchmarks.

Questions for authors
  1. Degraded-question validity: Are question answerability verified under each degraded condition? Please clarify how ambiguous or insufficiently visible cases are handled.

  2. Could the authors report explicit random-guessing baselines?

  3. Could the authors provide the complete SFT configuration, including optimization settings, trainable modules, input resolution, and clarify how these settings are matched across the ablations?

  4. Could the author provide a comparison regarding the scale and types of degradation between this benchmark and other similar benchmarks?

  5. The manuscript would benefit from careful proofreading. Beyond the unresolved “Figure ??” reference (line 070), there is a duplicated “Avg.” header in Table 2. Several typographical errors also remain, including “writting” (line 488), “discribe” (line 955), and “tamplates” (line 1187).

If the author can effectively solve the above problems, I am willing to raise my rating.

9 comments
AuthorsOpenReview ↗

Responses to Reviewer SWNG's W1

Dear reviewer SWNG, thank you for your professional and insightful comments on our work. Your feedback is valuable and plays as a crucial role in improving this paper. In the following, we address your questions one by one.

W1. Incremental gain from degradation-aware SFT

We agree that clean-only SFT already achieves most of the improvement over the pretrained model. The 0.1-point difference on original images indicates comparable performance under pristine conditions, while the 1.3-point difference on degraded images represents the additional advantage observed with degradation-aware supervision. To further clarify the unique advantage of degradation-aware SFT, we continue SFT from the existing checkpoints.

EpochClean-only SFTSpaceDG-SFTDelta
164.866.1+1.3
267.369.3+2.0

The continuation experiment provides clearer evidence for this incremental benefit. At the second-epoch checkpoints, SpaceDG-SFT improves by 3.2 points over epoch 1, compared with 2.5 points for clean-only SFT, increasing the observed advantage to 2.0 points. A paired bootstrap using 10,000 scene-level resamples of 1,102 questions from 320 scenes gives an unrounded gain of +1.97 points with a 95% CI of [+1.04, +2.92]. This supports a positive average advantage for SpaceDG-SFT and suggests that the benefit of degradation-aware SFT can grow with further training.

AuthorsOpenReview ↗

Responses to Reviewer SWNG's Q1

Q1. Answerability and ambiguity

Thank you for raising this concern. Our quality verification process assesses whether questions remain answerable and addresses potential ambiguity. During synthesis, we constrain degradation parameters to predefined ranges to avoid excessive visual corruption. A vision-language model then reviews each question and filters out those that are effectively unanswerable from the degraded images. Finally, human experts review all questions in SpaceDG-Bench and a sample of questions from SpaceDG-Dataset. Ambiguous or borderline cases are excluded.

AuthorsOpenReview ↗

Responses to Reviewer SWNG's Q2

Q2. Explicit random-guessing baselines

We report the expected accuracy of uniform random guessing over each question's available choices. The 680 multiple-choice questions include 585 four-option and 95 two-option questions, giving (585/4+95/2)/680=28.49%(585/4 + 95/2)/680 = 28.49\%. The two-option MCQs comprise 94 object-existence questions with lettered yes/no options and one camera-motion-direction question. The separate 24 yes/no questions require direct yes/no answers and have a chance accuracy of 50.00%. Averaging across all 704 fixed-choice questions gives 29.23%. Open-ended numerical questions are excluded because a random-guessing baseline is not applicable to these tasks, as noted in VSI-Bench [1].

MetricMultiple-choiceYes/noOverall
Random-guessing accuracy28.49%50.00%29.23%

[1] Jihan Yang et al. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. CVPR 2025.

Reviewer_k6yROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpaceDG, a benchmark for studying how visual degradation affects spatial reasoning in multimodal language models. The authors use 3D Gaussian Splatting to create degraded images of indoor scenes, covering nine different types of degradation, and test a range of existing models on the benchmark. They also fine-tune models using the generated data and observe improvements on both clean and degraded images. The work offers a useful way to examine where current models struggle when visual input quality degradations.

Critical strengths
  1. The paper studies an important but relatively underexplored question: how robust is spatial reasoning in MLLMs when the visual input is degraded? The evaluation shows a consistent drop in performance across the tested models. This makes the problem worth studying, particularly for systems that need to reason about spatial relationships from imperfect visual observations.

  2. The dataset is reasonably substantial. SpaceDG contains 971,090 QA instances from 584 indoor scenes, while SpaceDG-Bench contains 1,102 manually checked questions from 320 scenes. Each benchmark question is evaluated on clean images and under nine types of degradation, allowing comparisons across visual conditions.

  3. The evaluation also covers a broad range of models and spatial reasoning tasks. This helps reveal that degradation does not affect all aspects of spatial reasoning in the same way, and makes the benchmark useful for further investigation of spatial robustness.

Critical weaknesses
  1. It’s not clear how much of the gain really comes from the degradation-aware training. Fine-tuning helps a lot on both clean and degraded images, and the big jump on clean images suggests that just doing more spatial training might explain a good part of the improvement. Stronger controls would help — for example, clean-only fine-tuning and standard corruption augmentation with similar data size and compute. Checking performance on held-out degradations would also be useful.

  2. SpaceDG-Bench has 1,102 unique questions. Each one is tested under clean plus nine degradations, which gives 9,918 pairs. Those pairs aren’t independent, since the same question shows up multiple times. It would be better if the paper reported uncertainty in a way that takes this structure into account, for instance by bootstrapping over questions or scenes.

  3. There are a few presentation issues. Citation style looks inconsistent in places, some papers that are already published still only have arXiv links, and the introduction has a broken figure reference (“Figure ??”), etc..

Questions for authors
  1. Did you run comparisons against clean-only spatial fine-tuning and standard image corruption augmentation, keeping data size and compute roughly the same? How much of the extra gain actually comes from the 3DGS-based degradation method?

  2. Do the uncertainty estimates or significance tests take into account that the same questions appear under different conditions? For example, by doing bootstrap resampling at the question or scene level?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer k6yR's W1&Q1

Dear Reviewer k6yR, thank you for your time and effort in reviewing our manuscript and offering valuable suggestions. We would provide a detailed response to address your concerns.

W1&Q1. Clean-only SFT and evaluation on held-out degradations

Thank you for your questions. Table 6 includes a clean-only spatial SFT comparison, providing a reference for the gains associated with spatial-task training on clean images. We further extend both training runs into a second epoch. Clean-only SFT and SpaceDG-SFT achieve average scores of 67.3 and 69.3, respectively, across the nine degradations, widening the observed gap from 1.3 to 2.0 points. A 10,000-replicate paired bootstrap gives an unrounded difference of +1.97 points (scene-level 95% CI [+1.04, +2.92]), supporting a positive average advantage for SpaceDG-SFT over clean-only SFT. This result further motivates the hypothesis that, compared with clean-only SFT, degradation-aware SFT can use additional training exposure and computation more efficiently to improve spatial reasoning robustness.

EpochClean-only SFTSpaceDG-SFTDelta
164.866.1+1.3
267.369.3+2.0

For the suggested held-out evaluation, we conduct experiments in which one degradation category is excluded from training at a time. The table below reports the resulting models' scores on the excluded degradation types.

Category excluded from trainingTest degradationScore (%) ↑
Optical & dynamicDefocus64.0
Optical & dynamicDistortion69.9
Optical & dynamicMotion blur65.2
MeteorologicalHaze64.3
MeteorologicalWater droplets66.02
PhotometricLow light57.8
PhotometricOverexposure66.1
DigitalJPEG compression67.4
DigitalLow resolution63.8

For example, training without optical/dynamic degradations yields 64.0 on held-out defocus, compared with 63.2 for clean-only SFT. These results suggest that training on other degradation categories can promote robustness to unseen corruptions, although the degree of transfer varies across categories.

AuthorsOpenReview ↗

Responses to Reviewer k6yR's W2&Q2

W2&Q2. Paired uncertainty estimates

SpaceDG-Bench evaluates the same 1,102 questions on original images and under nine degradations. To preserve these pairs, we draw 320 scenes with replacement in each of 10,000 bootstrap replicates. Each sampled scene contributes all its questions and their scores across all conditions and models. We recompute each difference using the paper's question-weighted Accuracy/MRA score and take the central 95% of the replicate differences as its confidence interval. We also resample whole questions to check whether the choice of sampling unit changes the conclusions.

The representative models below span multiple model families and sizes, including general-purpose, spatial-intelligence, and robotic-brain models. All displayed recomputed differences are within 0.10 points of those calculated from the reported scores. For all 14 baselines, the average score across the nine degradations is lower than the original-image score, with both confidence intervals for the difference entirely below zero. SpaceDG-SFT scores 1.20 points higher than clean-only SFT on degraded images (scene-level 95% CI [+0.16, +2.26]).

ModelContrastPaper ΔRecomputed ΔScene-level 95% CIQuestion-level 95% CI
Kimi-VL-A3B-InstructDegraded − original−6.3−6.35[−8.27, −4.48][−8.22, −4.48]
Qwen3-VL-8B-InstructDegraded − original−7.0−6.91[−8.82, −5.06][−8.83, −4.99]
Qwen3-VL-32B-InstructDegraded − original−9.8−9.81[−11.67, −7.94][−11.69, −7.89]
Qwen3.5-9BDegraded − original−7.6−7.59[−9.80, −5.40][−9.71, −5.52]
SenseNova-SI-InternVL3-8BDegraded − original−4.6−4.53[−6.46, −2.55][−6.36, −2.71]
ACE-Brain-0-8BDegraded − original−5.1−5.14[−7.31, −2.98][−7.18, −3.03]
AuthorsOpenReview ↗

Responses to Reviewer k6yR's W3

W3. Presentation issues

Thank you for pointing out these issues. We have corrected the figure reference error and resubmitted the revised PDF. We will update the relevant references to cite the final published versions in a future revision.

Dear Reviewer k6yR,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As we're halfway through the discussion period, we would like to confirm whether our responses have effectively addressed your concerns. We have provided detailed responses to your concerns, and we hope they have adequately addressed your issues.

Best regards.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttals for AC

Dear Area Chair,

Thank you for handling our submission. We also appreciate all reviewers’ constructive feedback, which is invaluable in helping us improve our paper. Here, we provide a summary of our contributions and rebuttals for AC.

All three reviewers acknowledged the importance and value of studying spatial reasoning under visual degradation, an underexplored challenge relevant to real-world deployment. SpaceDG addresses this gap through physically grounded degradation synthesis, approximately 1M spatial QA pairs, and a human-verified benchmark covering nine degradation types and 11 spatial tasks. Our evaluation of 25 MLLMs reveals a consistent robustness gap and provides a resource for evaluating and improving spatial intelligence under imperfect observations.

We responded point by point to every weakness and question, providing additional experiments, statistical analyses, and methodological clarifications. All three reviewers acknowledged progress in addressing their concerns:

  • Reviewer SWNG confirmed that our responses addressed most concerns and raised the rating to 3 (Weak Acceptance). Our responses covered training-gain attribution, reproducibility, question answerability, random-guessing baselines, and benchmark comparisons.
  • Reviewer M5Rb acknowledged that the additional experiments addressed previous concerns to a reasonable extent and maintained the rating of 3 (Weak Acceptance). These additions included statistical validation, outdoor generalization, and CoT evaluation on two additional models.
  • Reviewer k6yR explicitly recognized the value of our work and accepted the paired bootstrap analysis. To address the remaining concerns about incremental gains and simpler alternatives, we further provided matched augmentation comparisons, analyses against real degradations, and a depth-aware synthesis alternative. These latest additions await further feedback; the reviewer expressed an intention to improve rating.

Thanks again for AC's and all reviewers' time and effort in reviewing our manuscript and offering valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.