acceptodds
Under review as a conference paper at ICLR 2027

Spatial Bias in Vision Language Models

Abstract

Spatial intelligence is essential for embodied tasks such as navigation, manipulation, and agentic interaction. Vision-Language Models (VLMs) increasingly exhibit spatial reasoning capabilities, yet priors learned from large-scale vision-language pre-training may override current visual evidence and induce systematic spatial errors. While social bias and visual hallucination have been widely studied, spatial bias remains underexplored. We define spatial bias as the systematic tendency of VLMs toward prior-aligned spatial predictions despite conflicting visual evidence. To investigate this phenomenon, we construct an original–counterfactual embodied visual question answering (VQA) benchmark covering diverse spatial tasks and prior sources, and introduce a multi-stage diagnostic framework that distinguishes counterfactual failures from prior-retention bias, language-aligned bias, and non-bias errors. We further investigate how input organization, contrastive context, background information, and visual guidance affect spatial bias, and evaluate multiple mitigation strategies. Experiments on diverse VLMs reveal widespread spatial bias that varies across model family, spatial task, prior source, and contextual conditions such as memory and visual information. Input and background configurations affect counterfactual robustness, while selective context suppression and explicit visual guidance can reduce bias. Prompt-based interventions yield limited and inconsistent gains, whereas supervised fine-tuning provides more reliable mitigation. These findings underscore the limitations of accuracy-only evaluation and the need to explicitly diagnose and mitigate spatial bias in embodied VLMs.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Model, Spatial Reasoning, Bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CYFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates spatial bias in vision-language models through a benchmark of 1,254 original–counterfactual image pairs covering six spatial tasks and four hypothesized prior sources. It introduces a behavioral diagnostic framework that separates original-answer retention, language-aligned errors, and residual errors. Experiments on 17 VLMs reveal substantial counterfactual performance drops.

Critical strengths
  1. The paired evaluation exposes failures that aggregate accuracy misses. Keeping questions and options fixed while changing spatial evidence provides a useful test of whether predictions update with the image.

  2. The experimental coverage is substantial. Evaluating 17 models and studying input context, visual guidance, and mitigation provides a useful characterization of counterfactual spatial robustness.

Critical weaknesses
  1. Are the results robust to answer-option ordering? The benchmark uses four-option questions, and the bias diagnosis relies on whether predictions match the original answer or the text-only answer. These measurements may therefore be affected by option-position preferences. Could the authors run circular evaluation by cyclically rotating all four options and consistently remapping the labels? Please report both accuracy and bias metrics across rotations, together with the proportion of samples answered correctly under every rotation. This would help distinguish stable spatial judgments from option-order effects.

  2. What is human performance on the benchmark? The paper reports model-based filtering and evaluation, but does not provide human accuracy or annotation-agreement results. These results are important for establishing whether the edited spatial relations are visually identifiable and the ground-truth answers are unambiguous. Could the authors evaluate independent human annotators on a representative subset of original–counterfactual pairs and report accuracy for each condition, inter-annotator agreement, and ambiguous-sample rates?

  3. Can prior conflict be separated from editing difficulty? Counterfactual images introduce both unusual spatial relations and visual edits. The performance drop may therefore reflect editing artifacts, changes in target appearance, or increased perceptual difficulty. Could the authors compare prior-conflicting edits against prior-consistent edits matched for editing magnitude, image quality, and target recognizability? An additional performance drop under prior conflict, with errors consistently favoring the conventional relation, would provide stronger evidence for the proposed prior-based explanation.

  4. Does answer agreement identify bias beyond incidental agreement? Retaining the original answer or matching a text-only prediction does not by itself establish that a spatial prior caused the error. The random baseline also produces a BR of approximately 37.5%. Could the authors compare the observed BR against a model-specific permutation baseline preserving answer frequencies? Additionally, do the main model comparisons and mitigation gains remain consistent on a fixed, shared eligible subset, given that eligibility depends on original-image correctness?

  5. Could the authors clarify the metric definition and computation costs? Table 2 defines mAcc using a logical OR, which measures whether at least one image in a pair is answered correctly. However, the values in Table 1 appear to use the arithmetic mean of original and counterfactual accuracy; for example, Qwen3-VL-8B has (0.665+0.283)/2=0.474(0.665 + 0.283)/2 = 0.474. In addition, Appendix D.1 states that no models are trained or fine-tuned, whereas Section 6.6 and Appendix D.3.2 report SFT experiments.

  6. Are cross-model bias comparisons robust to differences in eligible samples? The eligible set EME_M contains only samples that model MM answers correctly on the original image, so different models are evaluated on different subsets. A lower BR in Table 1 may therefore reflect differences in sample composition rather than a lower tendency toward spatial bias. Could the authors report comparisons on a shared eligible set, such as the intersection of samples answered correctly by the models being compared, together with its size and task distribution? Do the BR rankings remain consistent under this controlled comparison?

Questions for authors

please see the weaknesses.

4 comments

We sincerely thank the reviewer for the constructive suggestions. In response, we have conducted additional controlled experiments and robustness analyses, refined the metric definitions, and clarified the implementation details.

  1. Robustness to Answer-Option Ordering: We have added a robustness analysis in Appendix C.1.3 (Sec. “Robustness to Answer-Option Ordering,” Tab. 6–7), evaluating four cyclic option rotations across three representative models. We report accuracy and bias metrics, together with all-rotation correctness and prediction consistency. The results show that, despite some prediction-level sensitivity to option ordering, aggregate bias estimates remain relatively stable, suggesting that the observed bias cannot be explained solely by option-order effects.
  2. Human Performance: We have added a human evaluation in Appendix C.1.1 (Sec. “Human Validation and Annotation Reliability,” Tab. 3), involving three independent annotators on all original–counterfactual pairs. We report human accuracy, Fleiss' κ, annotator disagreement, and explicitly flagged ambiguity rates. The results demonstrate high human accuracy and strong inter-annotator agreement across both conditions, supporting the visual interpretability and annotation reliability of the benchmark. We also distinguish annotator disagreement from potential ambiguity to avoid overinterpreting annotation inconsistencies.
  3. Prior-Conflict vs. Prior-Consistent Editing: We have added a controlled comparison in Appendix C.1.5 (Sec. “Prior-Conflict vs. Prior-Consistent Editing,” Fig. 13 and Tab. 10). Both conditions edit the same target object while preserving the question and answer options, with prior-consistent edits retaining the original ground-truth answer. Despite comparable editing conditions and larger pixel-level changes in prior-consistent edits, prior-conflicting edits cause substantially greater performance degradation. These findings suggest that spatial prior conflict contributes to prediction errors beyond visual editing effects alone.
  4. Bias beyond Incidental Agreement: We have added model-specific permutation tests in Appendix C.1.7 (Sec. “Permutation Test for Bias Significance,” Tab. 12), using three null configurations that preserve marginal answer frequencies while disrupting sample-level associations. Across all 17 models, the observed BR significantly exceeds the permutation baselines (all p<0.001), with mean excess BR ranging from 34.35 to 41.26 pp. These results indicate that incidental answer agreement alone cannot explain the observed spatial bias.
  5. Metric definition and Computation Details: Thank you for pointing out these issues. We have corrected the definition of mAcc in Appendix B (Tab. 2) to the arithmetic mean of original and counterfactual accuracy, consistent with the reported results in Table 1. We have also clarified in Appendix D.1 that the inference-only setting applies to baseline evaluations, whereas the SFT experiments in Sec. 6.6 and Appendix D.3.2 were conducted separately as mitigation studies. We apologize for the omission and have supplemented the computational details, including GPU resources, fine-tuning configurations, and recorded elapsed times, while distinguishing evaluation and SFT costs.
  6. BR on Shared Eligible Subset: We have added a shared-eligibility analysis in Appendix C.1.6, recomputing BR on the intersection of samples correctly answered by all 17 models. We also report the subset composition across all six spatial tasks and four bias-reason categories. Although all categories are represented, we acknowledge that the shared subset is not distribution-matched to the full benchmark, with physical–geometric principles overrepresented (58.62% vs. 24.56%). The overall BR rankings remain strongly correlated with the original rankings (Spearman's ρ = 0.877; Kendall's τ = 0.746), with the lowest- and highest-BR models retaining their positions. Additional within-group comparisons yield similarly high correlations (ρ = 0.917–1.000). However, some intermediate rankings change, and bootstrap confidence intervals overlap for all adjacent model pairs. We have therefore clarified that the results support broad ranking consistency under shared eligibility, rather than definitive pairwise ordering.

Thank you for the additional experiments. The editing control is helpful, but prior-consistent edits retain the original answer, whereas prior-conflicting edits require an answer change. This leaves prior conflict confounded with answer-change requirements: a model that simply preserves its original prediction would also show the reported performance difference. Could the authors clarify whether the gap persists when both prior-consistent and prior-conflicting edits change the ground-truth answer, with comparable perceptual difficulty?

Thank you for your reply. We are currently running the additional control experiment you suggested, which involves prior-consistent edits that alter the ground-truth answer. Since this experiment requires constructing new data, it may take some additional time. We are doing our best to complete it as soon as possible. We appreciate your patience and understanding, and we hope these additional results will adequately address your concerns.

Thank you for this constructive suggestion. In response, we have conducted additional experiments with matched answer-change requirements to further disentangle the effects of prior conflict from those of image editing and answer changes. Our detailed responses and the corresponding experimental results are presented below:

We have extended the controlled editing study in Appendix C.1.5 (Sec. “Prior-Consistent vs. Prior-Conflicting Editing,” Fig. 13 and Tab. 10) by introducing three editing conditions: (1) invariant prior-consistent (ICons.), which preserves both the conventional spatial relation and the original ground-truth answer; (2) flipped prior-consistent (FCons.), which changes the ground-truth answer while maintaining consistency with common-sense spatial priors; and (3) prior-conflicting (Conf.), which changes the ground-truth answer in a direction that contradicts common-sense spatial priors. All three conditions modify the same target object while keeping the question and answer options unchanged.

The results show that both ICons. and FCons. achieve higher accuracy than Conf. across seven representative models, indicating that prior conflict contributes to prediction errors beyond the effects associated with image editing and answer changes. Notably, FCons. outperforms Conf. by 7.1–28.9 pp, demonstrating that the performance gap persists under matched answer-change requirements. Meanwhile, FCons. generally yields lower accuracy than ICons., suggesting additional challenges associated with image editing and answer changes. Further model-level comparisons reveal relatively smaller FCons.–Conf. accuracy gaps for the evaluated Claude models, an increasing gap with model scale in the Qwen3-VL series, and noticeable prior-conflict sensitivity in spatial-specialist models. These gaps provide an indication of each model's sensitivity to conflicting spatial priors under matched answer-change requirements.

We sincerely hope that our responses and additional experiments have adequately addressed your concerns. If you find the new experiments and clarifications helpful, we would greatly appreciate it if you could consider increasing your score accordingly. We have made every effort to address your concerns through additional data construction, validation, and extensive experimental testing, which required considerable additional time and effort from our team. We sincerely hope that these additional efforts will meet with your approval. We truly appreciate the time and effort you have devoted to reviewing our paper and providing constructive feedback. Thank you again for your consideration, and we wish you all the best.

Reviewer_H1CfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether VLMs can still retain similar spatial answers after the relevant visual evidence is changed. To test this, the authors constructed 1,254 original–counterfactual image pairs for spatial tasks. In each pair, the question and four options in the prompt remain unchanged, while the edited image changes the target's position, orientation, depth, or scene layout. The paper proposes several methods and a staged diagnostic for matching counterfactual answers and text-only answers. It evaluates 17 VLMs across 6 spatial task types, studies input order and background information, and compares prompts fine-tuned with LoRA models. The main empirical answer is very clear: large models often lose accuracy after spatial changes, and paired supervision is more reliable than prompt correction in reducing failures.

Critical strengths

The strongest aspect of this paper is the paired dataset. Keeping the question and options unchanged while changing one spatial factor makes the model's incorrect judgments easy to detect and the evaluation easy to interpret. Its coverage is also broad: 17 models, 6 task types, and several kinds of spatial changes. This is more informative than a single aggregate reasoning score.

This empirical study is broader than an ordinary dataset paper. The random baseline methods, comparisons by input ordering, background controls, prompt results, and paired model fine-tuning experiments show that the problem is not limited to one model or one prompt. The paper also states the limitations of its diagnostic analysis rather than concealing them.

Critical weaknesses

The paper treats retention of answers on counterfactual images, or agreement with text-only answers, as evidence of spatial bias. Although this is a useful behavioral definition, it does not by itself prove that a learned spatial prior causes the errors. Editing artifacts, harder counterfactual images, answer-option learning effects, or simply failure to recognize the edit could all produce the same outputs. The white-box analysis is a good attempt to go further, but it is tested only on Qwen3-VL-8B and only on the subset with the required measurement data. It would be better if the paper described "spatial bias" as a behavioral characteristic of this benchmark, or added human checks and stronger controls over the experimental variables before making broader causal claims.

I think the data filtering issue is more serious. The authors removed 43 candidate pairs when all three curation models answered the original image incorrectly but answered the counterfactual image correctly, because this violated the expected assumption that "the original image should be easier to judge." This decision was based on model predictions on the evaluation pool and happened to remove exactly the cases where the counterfactuals were not difficult. It may change the difficulty balance and affect CER, paired accuracy, and the derived bias rate. The paper should report the main results before this filtering, or at least present the filtered and unfiltered sets side by side. The lack of human-verified edited images and the small 251-pair test set for SFT make the later large mitigation effects harder to interpret.

Questions for authors
  1. Were the edited images and their labels checked by human annotators? If so, how many pairs were checked, and what was the agreement rate? If not, what evidence is there that these edits are unambiguous to humans, not just to the construction models?

  2. Can the authors report all main metrics before applying the 43 model-based filtered pairs? This result would show how much the reported bias rate depends on the assumption that "the original image must be easier."

  3. The four prior-source labels are described as hypotheses about why errors occur. Is there evidence that the errors differ in the proposed cultural, physical, semantic, and reference-frame ways, or should these labels be presented only as constructed categories?

  4. How stable are the SFT results across different random seeds and training/test splits? The test set contains 251 pairs, so variance estimates would help assess the reported 32.7 percentage point improvement in counterfactual accuracy.

2 comments

We sincerely thank the reviewer for the thoughtful feedback. We have carefully addressed the concerns through additional experiments, independent human validation, and revisions to the manuscript.

  1. Human Annotation: We have added a comprehensive human validation study in Appendix C.1.1 (Sec. “Human Validation and Annotation Reliability,” Table 3), where three independent annotators evaluated all 1,254 original–counterfactual pairs (2,508 images) without access to ground-truth labels. Human accuracy reaches 92.88% on original images and 93.04% on counterfactual images, with high inter-annotator agreement (Fleiss' κ = 0.866/0.858). Only 1.52% of pairs were flagged as potentially ambiguous by a majority of annotators. These results support the reliability of the annotations and indicate that counterfactual edits remain visually interpretable to humans.
  2. Data Filtering: The filtering was intended to focus the benchmark on prior reliance rather than general recognition failures, consistent with our diagnostic framework. To further verify the robustness of our findings, we conduct an additional sensitivity analysis by including the 43 previously excluded pairs and recomputing all metrics for 17 models on the full 1,297-pair pool. As reported in Appendix C.1.4 (Table 9 and Fig. 12), BR changes by only −0.54 to +0.08 percentage points (mean −0.23 pp), with unchanged model rankings. Other main metrics also show limited variation, confirming that our findings are robust to the filtering procedure.
  3. Prior-source Labels: The four categories are used to construct counterfactual samples targeting different types of spatial priors. Our experiments show that model errors vary across these categories. We have clarified their role as hypothesis-driven construction categories in Section 2.
  4. Stability of the SFT Results: We have added a stability analysis in Appendix C.1.8 (Table 13), comprising SFT runs across three models, three training seeds, and three data partitions. On the original 251-pair test set, Qwen3-VL achieves an average counterfactual accuracy gain of 30.01 ± 2.04 pp across seeds, while SpatioLM and InternVL3.5 achieve 20.98 ± 1.28 pp and 22.71 ± 1.83 pp, respectively. The improvements persist across alternative splits, including an enlarged 377-pair test set. We further report paired source-group bootstrap 95% confidence intervals using 10,000 resamples, all of which exclude zero. These results support the consistency of the SFT gains across the evaluated configurations.
  5. Bias Evaluation: Our spatial bias evaluation follows the counterfactual assessment paradigm adopted in Vision Language Models Are Biased (ICLR 2026), which measures bias through prior-consistent responses to counterfactual images. Our diagnostic framework is also inspired by HallusionBench (CVPR 2024, Fig. 3), which uses paired prediction outcomes to characterize model failure patterns. Building on these established approaches, we investigate spatial bias through original–counterfactual comparisons and complementary text-only diagnostics. We further provide human validation, controlled counterfactual comparisons, and white-box analysis to strengthen the interpretation of the observed patterns. We have clarified the methodological foundations and added the corresponding references in Sec. 4 (Spatial Bias Diagnosis). We also explicitly characterize spatial bias as an observable behavioral signature consistent with prior reliance, while identifying finer-grained attribution of perception, cross-modal alignment, and reasoning failures as an important direction for future work (Appendix E).

Dear Reviewer, I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I want to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are important to us, and we're eager to address any remaining issues to improve our work. Thank you for your time and effort in reviewing our paper.

Reviewer_NynwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates spatial bias in VLMs by introducing a counterfactual benchmark comprising paired original and spatially modified images across diverse spatial reasoning tasks and potential bias sources. The authors propose a multi-stage diagnostic framework that distinguishes prior-retention bias, language-aligned bias, and other spatial errors beyond conventional accuracy-based evaluation.

Through extensive experiments across 17 VLMs, the paper reveals variations in spatial bias across model families and spatial tasks. Furthermore, systematic analyses of input configurations, background configurations, and spatial bias mitigation strategies provide additional insights into how contextual and visual information influence spatial bias, as well as the effectiveness of prompting and supervised fine-tuning in mitigating it.

Critical strengths
    1. A valuable counterfactual benchmark for spatial reasoning
      The paper introduces a counterfactual benchmark that evaluates whether VLMs can adapt their spatial predictions when task-relevant visual evidence changes. Such counterfactual evaluation remains relatively underexplored in existing spatial VLM benchmarks. By constructing paired original and counterfactual scenes, the benchmark provides a useful testbed for assessing the robustness of spatial reasoning and identifying systematic spatial biases.
    1. Clear and informative empirical analyses
      The experiments in Section 6 systematically investigate the effects of input configurations, background configurations, and spatial bias mitigation strategies. These analyses are straightforward and clearly presented, providing useful insights into the spatial reasoning behaviors of VLMs and how different contextual and visual interventions influence their predictions.
Critical weaknesses

Insufficient reproducibility and quality validation of the counterfactual benchmark. Although the paper provides general construction guidelines, prompt templates, and representative examples in Appendix F, several essential details remain insufficiently specified:

  • Data construction: How are source images, target objects, and counterfactual spatial states systematically selected across different task–prior categories? How are questions and distractor options generated, and what procedures ensure their validity?
  • Counterfactual image editing: While the paper describes annotation-guided spatial transformations, the specific image editing models, tools, and implementation procedures remain unclear. How do the authors ensure that unintended visual changes or editing artifacts do not affect model predictions?
  • Quality validation: The paper describes several quality-control checks and a model-based filtering procedure. However, the implementation and quantitative outcomes of these checks are insufficiently documented. In particular, removing samples for which all three verification models fail on the original image but succeed on the counterfactual image does not independently establish the visual quality or annotation correctness of the remaining samples. A systematic human evaluation of intervention correctness, visual quality, and answer validity would strengthen the benchmark's reliability.

Since the counterfactual benchmark is a central contribution, the insufficiently specified implementation details and limited quantitative evidence of quality validation raise substantial concerns about its reproducibility and reliability. If these aspects have already been addressed in the paper, I would appreciate it if the authors could clarify them and point me to the relevant sections.

Questions for authors

Clarification on benchmark construction and annotation quality.

My primary concern is the reliability of the counterfactual benchmark. For example, in Figure 3 (Spatial Action Bias), the counterfactual answer, “The car is turning left via the dedicated left-turn lane,” is marked as incorrect. However, based on my interpretation of the counterfactual image, this answer appears consistent with the depicted scene. Could the authors clarify why this answer is considered incorrect and provide the corresponding ground-truth annotation? More broadly, what procedures were used to verify the correctness of counterfactual edits, questions, and answer labels across the benchmark?

2 comments

We sincerely thank the reviewer for the detailed comments. We have expanded Appendix F and C to improve the reproducibility and quality validation of our benchmark.

  1. Data Construction: We provide detailed construction procedures and complete raw-data examples in Appendix F.3 ( Listing 1-3), including source images and annotations, target selection, spatial interventions, question–answer pairs, editing tools and prompts, and explicit quality-control checks. For instance, the released record SPC_002 documents the entire construction process, from the original VisDrone annotation to the counterfactual edit and its verification results.
  2. Counterfactual Image Editing: We have added Appendix F.1.2 (“Counterfactual Image Editing and Quality Control”) to specify the image-editing tools and implementation procedures. Counterfactual images are constructed using OpenAI's image editor, annotation-guided deterministic transformations implemented with Pillow/SciPy, and manual editing with Photoshop. Editing instructions explicitly constrain modifications to the target spatial attributes while preserving unrelated objects, scene layout, and visual appearance. To ensure editing quality, we apply consistency checks covering target identifiability, intended spatial changes, preservation of unrelated scene content, and answer-label validity, supported by human verification. We have also provided complete construction records in Appendix F.3 (Listing 1-3), including editing tools, prompts, source annotations, image dimensions, and itemized quality-control results. These additions clarify how unintended visual changes are controlled and make the editing process more transparent and reproducible.
  3. Quality Validation: To further establish benchmark reliability, we conduct an independent human evaluation of all original–counterfactual pairs with three annotators. As reported in Appendix C.1.1 (Sec. “Human Validation and Annotation Reliability,” Table 3), human accuracy reaches 92.88%/93.04% on original/counterfactual images, with high inter-annotator agreement (Fleiss' κ = 0.866/0.858). Only 1.52% of pairs were flagged as potentially ambiguous by a majority of annotators. We also provide a filtering sensitivity analysis in Appendix C.1.4 (Table 9 and Fig. 12), showing that including the 43 excluded pairs changes BR by at most 0.54 pp, without affecting model rankings.
  4. Question: Regarding the spatial action sample shown in Fig. 3, we confirm that the ground-truth answer is “U-turn,” independently selected by all three human annotators. The distinction from a left turn depends on the red car's position relative to the small planted traffic island: turning before passing the island indicates a U-turn, whereas turning after passing it would correspond to a left turn in this scene. Both actions are included among the answer options, requiring fine-grained spatial interpretation.

Dear Reviewer, I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I want to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are important to us, and we're eager to address any remaining issues to improve our work. Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.