- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates geographic bias in remote-sensing vision-language models (VLMs), focusing on when geographic priors lead to predictions unsupported by visual evidence. The authors propose a four-stage framework (Audit, Ground, Trace, and Steer) that uses pixel-identical geographic counterfactuals to identify geographic influence, external task-specific evidence to assess bias, activation patching to localize causal representations, and inference-time steering to mitigate harmful effects without retraining. Experiments across four VLMs on socioeconomic assessment (fMoW/RWI) and disaster-damage assessment (xBD) demonstrate substantial geographic sensitivity. The proposed method reduces counterfactual prediction gaps by approximately 65–82% on held-out images, outperforming conventional activation steering while largely preserving task utility.
Critical strengths
- The paper addresses an important problem in remote-sensing VLMs by distinguishing geographic sensitivity from potentially harmful bias. The pixel-identical counterfactual design, combined with task-specific external grounding, provides a systematic way to investigate geographic influence while controlling for visual differences. This is a meaningful contribution to evaluating VLM reliability.
- The proposed method combines causal activation patching with harm-weighted steering and demonstrates consistent improvements across four VLMs and two tasks. The extensive ablations and comparisons with standard activation steering (Tables 3–4) provide convincing evidence of its effectiveness, supporting the paper's practical significance.
Critical weaknesses
- While pixel-identical counterfactuals isolate geographic influence, prediction changes do not necessarily indicate unjustified bias, particularly when geographic priors provide legitimate contextual information. Although external grounding is introduced (Section 3, Table 2), the criteria for distinguishing harmful bias from useful geographic reasoning remain insufficiently established, weakening the central claim.
- The proposed steering method reduces prediction differences relative to cue-absent representations, but this does not necessarily demonstrate selective removal of harmful geographic information. Although aggregate task utility is largely preserved (Tables 3–4), stronger justification is needed to establish that beneficial geographic information is retained rather than broadly suppressed.
Questions for authors
- How do the authors distinguish harmful geographic bias from legitimate geographic priors when identical visual evidence may still support different predictions under different geographic contexts? To what extent does external grounding establish that the observed geographic influence is unjustified?
- Since the steering method explicitly reduces deviations from cue-absent predictions, what evidence demonstrates that it selectively removes harmful geographic influence rather than suppressing geographic information more broadly? In particular, how do the authors interpret the preservation of beneficial geographic priors beyond aggregate task-utility metrics?
Rebuttal Summary and Appreciation to the Area Chairs and Reviewers
We sincerely thank the Area Chairs and all reviewers for their thoughtful evaluations, constructive feedback, and positive recognition of our work. We particularly appreciate that the reviewers highlighted the importance, novelty, and practical significance of investigating when geographic priors become harmful biases in remote-sensing VLMs. We are also grateful for their recognition of our controlled counterfactual design, causal intervention framework, and extensive experimental validation.
We have carefully addressed all reviewer concerns in our individual responses. Below, we summarize the main clarifications and supporting evidence:
Methodological Novelty: Our contribution goes beyond conventional activation patching and steering by integrating causal localization with independently grounded harm weighting. Our method achieves 73.3% mitigation, compared with 38.5% for standard CAA, with controlled ablations confirming the contributions of both components.
Grounded Utility and Selective Mitigation: We distinguish harmful geographic influence from useful geographic priors using independent task evidence. On an expanded evaluation of 600 sites across 24 countries, steering improves RWI correlation to 0.36, compared with 0.30 without geographic information. Additional paired confidence intervals support these improvements.
Robustness and Generalization: Our experiments demonstrate consistent performance across four VLMs, multiple geographic regions, alternative prompt formulations, and unseen country-level cues. Additional repeated-subset evaluations further support the stability of the learned steering directions.
We greatly appreciate the reviewers' constructive reviews. We believe our detailed responses and additional supporting evidence address the main concerns while reinforcing the novelty, soundness, and practical significance of the proposed framework. We respectfully hope the reviewers will consider updating their scores in light of these clarifications, and we thank the Area Chairs for their careful consideration of our submission.