- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates spatial bias in vision-language models through a benchmark of 1,254 original–counterfactual image pairs covering six spatial tasks and four hypothesized prior sources. It introduces a behavioral diagnostic framework that separates original-answer retention, language-aligned errors, and residual errors. Experiments on 17 VLMs reveal substantial counterfactual performance drops.
Critical strengths
-
The paired evaluation exposes failures that aggregate accuracy misses. Keeping questions and options fixed while changing spatial evidence provides a useful test of whether predictions update with the image.
-
The experimental coverage is substantial. Evaluating 17 models and studying input context, visual guidance, and mitigation provides a useful characterization of counterfactual spatial robustness.
Critical weaknesses
-
Are the results robust to answer-option ordering? The benchmark uses four-option questions, and the bias diagnosis relies on whether predictions match the original answer or the text-only answer. These measurements may therefore be affected by option-position preferences. Could the authors run circular evaluation by cyclically rotating all four options and consistently remapping the labels? Please report both accuracy and bias metrics across rotations, together with the proportion of samples answered correctly under every rotation. This would help distinguish stable spatial judgments from option-order effects.
-
What is human performance on the benchmark? The paper reports model-based filtering and evaluation, but does not provide human accuracy or annotation-agreement results. These results are important for establishing whether the edited spatial relations are visually identifiable and the ground-truth answers are unambiguous. Could the authors evaluate independent human annotators on a representative subset of original–counterfactual pairs and report accuracy for each condition, inter-annotator agreement, and ambiguous-sample rates?
-
Can prior conflict be separated from editing difficulty? Counterfactual images introduce both unusual spatial relations and visual edits. The performance drop may therefore reflect editing artifacts, changes in target appearance, or increased perceptual difficulty. Could the authors compare prior-conflicting edits against prior-consistent edits matched for editing magnitude, image quality, and target recognizability? An additional performance drop under prior conflict, with errors consistently favoring the conventional relation, would provide stronger evidence for the proposed prior-based explanation.
-
Does answer agreement identify bias beyond incidental agreement? Retaining the original answer or matching a text-only prediction does not by itself establish that a spatial prior caused the error. The random baseline also produces a BR of approximately 37.5%. Could the authors compare the observed BR against a model-specific permutation baseline preserving answer frequencies? Additionally, do the main model comparisons and mitigation gains remain consistent on a fixed, shared eligible subset, given that eligibility depends on original-image correctness?
-
Could the authors clarify the metric definition and computation costs? Table 2 defines mAcc using a logical OR, which measures whether at least one image in a pair is answered correctly. However, the values in Table 1 appear to use the arithmetic mean of original and counterfactual accuracy; for example, Qwen3-VL-8B has . In addition, Appendix D.1 states that no models are trained or fine-tuned, whereas Section 6.6 and Appendix D.3.2 report SFT experiments.
-
Are cross-model bias comparisons robust to differences in eligible samples? The eligible set contains only samples that model answers correctly on the original image, so different models are evaluated on different subsets. A lower BR in Table 1 may therefore reflect differences in sample composition rather than a lower tendency toward spatial bias. Could the authors report comparisons on a shared eligible set, such as the intersection of samples answered correctly by the models being compared, together with its size and task distribution? Do the BR rankings remain consistent under this controlled comparison?
Questions for authors
please see the weaknesses.
4 comments
Thank you for the additional experiments. The editing control is helpful, but prior-consistent edits retain the original answer, whereas prior-conflicting edits require an answer change. This leaves prior conflict confounded with answer-change requirements: a model that simply preserves its original prediction would also show the reported performance difference. Could the authors clarify whether the gap persists when both prior-consistent and prior-conflicting edits change the ground-truth answer, with comparable perceptual difficulty?
Thank you for your reply. We are currently running the additional control experiment you suggested, which involves prior-consistent edits that alter the ground-truth answer. Since this experiment requires constructing new data, it may take some additional time. We are doing our best to complete it as soon as possible. We appreciate your patience and understanding, and we hope these additional results will adequately address your concerns.
Thank you for this constructive suggestion. In response, we have conducted additional experiments with matched answer-change requirements to further disentangle the effects of prior conflict from those of image editing and answer changes. Our detailed responses and the corresponding experimental results are presented below:
We have extended the controlled editing study in Appendix C.1.5 (Sec. “Prior-Consistent vs. Prior-Conflicting Editing,” Fig. 13 and Tab. 10) by introducing three editing conditions: (1) invariant prior-consistent (ICons.), which preserves both the conventional spatial relation and the original ground-truth answer; (2) flipped prior-consistent (FCons.), which changes the ground-truth answer while maintaining consistency with common-sense spatial priors; and (3) prior-conflicting (Conf.), which changes the ground-truth answer in a direction that contradicts common-sense spatial priors. All three conditions modify the same target object while keeping the question and answer options unchanged.
The results show that both ICons. and FCons. achieve higher accuracy than Conf. across seven representative models, indicating that prior conflict contributes to prediction errors beyond the effects associated with image editing and answer changes. Notably, FCons. outperforms Conf. by 7.1–28.9 pp, demonstrating that the performance gap persists under matched answer-change requirements. Meanwhile, FCons. generally yields lower accuracy than ICons., suggesting additional challenges associated with image editing and answer changes. Further model-level comparisons reveal relatively smaller FCons.–Conf. accuracy gaps for the evaluated Claude models, an increasing gap with model scale in the Qwen3-VL series, and noticeable prior-conflict sensitivity in spatial-specialist models. These gaps provide an indication of each model's sensitivity to conflicting spatial priors under matched answer-change requirements.
We sincerely hope that our responses and additional experiments have adequately addressed your concerns. If you find the new experiments and clarifications helpful, we would greatly appreciate it if you could consider increasing your score accordingly. We have made every effort to address your concerns through additional data construction, validation, and extensive experimental testing, which required considerable additional time and effort from our team. We sincerely hope that these additional efforts will meet with your approval. We truly appreciate the time and effort you have devoted to reviewing our paper and providing constructive feedback. Thank you again for your consideration, and we wish you all the best.
We sincerely thank the reviewer for the constructive suggestions. In response, we have conducted additional controlled experiments and robustness analyses, refined the metric definitions, and clarified the implementation details.