acceptodds
Under review as a conference paper at ICLR 2027

When Geographic Priors Become Bias: Causal Steering in Remote-Sensing VLMs

Abstract

Geographic context can be a useful prior in remote sensing, but it becomes problematic when it overrides fixed visual evidence. We study this transition from geographic prior to geographic bias in remote-sensing vision-language models (VLMs) using pixel-identical geographic counterfactuals, where only the stated location is changed while the visual input remains unchanged. This isolates geographic influence from genuine differences in landscape, infrastructure, or physical damage. Across socioeconomic assessment and disaster-damage estimation, we find that geographic cues can substantially shift predictions even when the underlying imagery is identical, with external RWI and damage annotations revealing when such shifts are unsupported. We then use activation patching to identify transformer representations that causally mediate these effects and construct bias-weighted steering directions for inference-time intervention. Steering consistently reduces geographic counterfactual gaps on unseen images while preserving task-relevant utility. To the best of our knowledge, this is the first framework to jointly audit, causally localize, and selectively steer geographic bias in remote-sensing VLMs under controlled pixel-identical counterfactuals. Our results show that geographic priors can become causal shortcuts, but that these effects can be systematically identified and mitigated without retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
remote sensing, vision-language models, geographic bias, counterfactual auditing, causal tracing, activation steering

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hhvAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates geographic bias in remote-sensing vision-language models (VLMs), focusing on when geographic priors lead to predictions unsupported by visual evidence. The authors propose a four-stage framework (Audit, Ground, Trace, and Steer) that uses pixel-identical geographic counterfactuals to identify geographic influence, external task-specific evidence to assess bias, activation patching to localize causal representations, and inference-time steering to mitigate harmful effects without retraining. Experiments across four VLMs on socioeconomic assessment (fMoW/RWI) and disaster-damage assessment (xBD) demonstrate substantial geographic sensitivity. The proposed method reduces counterfactual prediction gaps by approximately 65–82% on held-out images, outperforming conventional activation steering while largely preserving task utility.

Critical strengths
  1. The paper addresses an important problem in remote-sensing VLMs by distinguishing geographic sensitivity from potentially harmful bias. The pixel-identical counterfactual design, combined with task-specific external grounding, provides a systematic way to investigate geographic influence while controlling for visual differences. This is a meaningful contribution to evaluating VLM reliability.
  2. The proposed method combines causal activation patching with harm-weighted steering and demonstrates consistent improvements across four VLMs and two tasks. The extensive ablations and comparisons with standard activation steering (Tables 3–4) provide convincing evidence of its effectiveness, supporting the paper's practical significance.
Critical weaknesses
  1. While pixel-identical counterfactuals isolate geographic influence, prediction changes do not necessarily indicate unjustified bias, particularly when geographic priors provide legitimate contextual information. Although external grounding is introduced (Section 3, Table 2), the criteria for distinguishing harmful bias from useful geographic reasoning remain insufficiently established, weakening the central claim.
  2. The proposed steering method reduces prediction differences relative to cue-absent representations, but this does not necessarily demonstrate selective removal of harmful geographic information. Although aggregate task utility is largely preserved (Tables 3–4), stronger justification is needed to establish that beneficial geographic information is retained rather than broadly suppressed.
Questions for authors
  1. How do the authors distinguish harmful geographic bias from legitimate geographic priors when identical visual evidence may still support different predictions under different geographic contexts? To what extent does external grounding establish that the observed geographic influence is unjustified?
  2. Since the steering method explicitly reduces deviations from cue-absent predictions, what evidence demonstrates that it selectively removes harmful geographic influence rather than suppressing geographic information more broadly? In particular, how do the authors interpret the preservation of beneficial geographic priors beyond aggregate task-utility metrics?
Reviewer_pgM3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose to test remote-sensing VLMs by keeping the image identical and changing only the stated location, such as “Africa” versus “Europe.” They find that this change alone can shift outcomes (in their experiments socioeconomic and disaster-damage predictions); the authors argue that this can introduce bias in models, if geographic cues harm predictive performance. The authors propose an activation patching approach to find late layers causing this effect, which allows them to subtract a steering direction at inference time. This approach reduces the geographic counterfactual gap with roughly equal performance.

Critical strengths

I want to highlight the following strengths of the submission

  • The paper poses a very interesting question! The idea of disentangling when geographic features vs data features dominate a prediction in a geo-prior setting is super interesting. Pairing that with the follow-up question of when geographic context might make models worse / bias them, has several practical implications and is also fascinating.
  • The experiments, especially the illustration in Fig 2, are well suited to highlight the idea of the paper. However, they could be a bit more extensive.
Critical weaknesses

The submission has the following shortcomings, ordered roughly from, in my opinions, most to least severe:

  • I found the submission to be kind of hard to follow / understand; some passages read very AI-written in their phrasing; I felt the methodological section was in parts unnecessarily mathy and dense. The core idea of the paper is rather simple - which is a good thing! I wished the authors embraced this more and simplified the writing and methodological presentation.
  • The figures in the paper are basically illegible (way too small); you have to zoom in a lot in order to be able to see details. If you're printing out the paper to read it you could simply not understand the figures. Visualizations can really help getting the idea of this paper across. Figure 2 is a great concept, but the execution is not really helping it shine. Figures should be heavily simplified and reworked with a focus on legibility.
  • The paper does not contextualize itself within the existing work on geographic priors enough. They should be explained more, and it should be highlighted where they were found to be useful. For example, the original geo-prior paper is not even cited. [1]
  • I am a bit confused about the xBD experiment. xBD does not have any Africa events right? So isn't that experiment a bit besides the point? And where does the Africa data from tables 11B / 13A come from then?
  • 'We find that geographic cues can substantially shift predictions even when the underlying imagery is identical,' from the abstract is not very useful. Thats the whole idea of a geo-prior and, as the authors themselves say, not an issue per-se. I'd rewrite this passage in the abstract.

References:

[1] Mac Aodha, Oisin, Elijah Cole, and Pietro Perona. "Presence-only geographical priors for fine-grained image classification." ICCV 2019

Questions for authors

Please see my shortcomings above; particularly, I'd like to know about the xBD experiments, which to my knowledge, don't include Africa data.

Reviewer_7fHvOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how geographic cues affect remote-sensing vision-language models when the imagery remains unchanged. It combines controlled geographic prompt changes with activation patching to identify influential layers, then constructs task-error-weighted steering vectors to reduce harmful geographic influence without updating model parameters. Reported experiments on socioeconomic and disaster-damage assessment show reduced sensitivity to geographic cues while largely preserving task performance.

Critical strengths

1.The controlled geographic prompt swaps provide a clear way to isolate the influence of stated location while keeping visual evidence fixed, avoiding confounding from differences between scenes.

2.The intervention keeps pretrained model parameters frozen. Comparisons with standard CAA and component ablations support the contribution of both causal layer selection and task-error-based weighting.

Critical weaknesses

1.The methodological novelty is limited. Activation patching and contrastive activation steering are established techniques. The main algorithmic additions are intervention-based layer selection and task-error-based weighting of activation differences. These are useful refinements, but the contribution is primarily their integration and application to remote sensing.

2.The practical advantage over simply removing geographic cues is modest. In Table 11, removing the cue achieves disaster MAE of 0.188, compared with 0.190 for an incorrect cue followed by steering. For socioeconomic assessment, RWI correlation improves only from 0.29 without geography to 0.30 with steering. The large reductions in counterfactual gaps therefore translate into relatively small gains in grounded task performance.

3.A key dataset provenance discrepancy affects the useful-prior preservation results. The paper reports 300 authentic African disaster examples, but the described 19-event xBD release contains no African events in its original event list. The source and event identities of these examples must be clarified before the corresponding results can be reliably assessed.

Questions for authors

1.Which disaster events and dataset release provide the 300 authentic African examples? Please provide event names and sample IDs, and reconcile this subset with the described 19-event xBD release, whose event list contains no African events.

2.Can you report paired task-performance differences, with confidence intervals, between steering and simply removing geographic cues, separately for authentic and incorrect cues? This would clarify the practical benefit beyond reducing counterfactual gaps.

3.How sensitive are the socioeconomic steering directions and performance gains to the choice of the 32 RWI-matched training sites? Can you report results across multiple training subsets or random seeds?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Appreciation to the Area Chairs and Reviewers

We sincerely thank the Area Chairs and all reviewers for their thoughtful evaluations, constructive feedback, and positive recognition of our work. We particularly appreciate that the reviewers highlighted the importance, novelty, and practical significance of investigating when geographic priors become harmful biases in remote-sensing VLMs. We are also grateful for their recognition of our controlled counterfactual design, causal intervention framework, and extensive experimental validation.

We have carefully addressed all reviewer concerns in our individual responses. Below, we summarize the main clarifications and supporting evidence:

Methodological Novelty: Our contribution goes beyond conventional activation patching and steering by integrating causal localization with independently grounded harm weighting. Our method achieves 73.3% mitigation, compared with 38.5% for standard CAA, with controlled ablations confirming the contributions of both components.

Grounded Utility and Selective Mitigation: We distinguish harmful geographic influence from useful geographic priors using independent task evidence. On an expanded evaluation of 600 sites across 24 countries, steering improves RWI correlation to 0.36, compared with 0.30 without geographic information. Additional paired confidence intervals support these improvements.

Robustness and Generalization: Our experiments demonstrate consistent performance across four VLMs, multiple geographic regions, alternative prompt formulations, and unseen country-level cues. Additional repeated-subset evaluations further support the stability of the learned steering directions.

We greatly appreciate the reviewers' constructive reviews. We believe our detailed responses and additional supporting evidence address the main concerns while reinforcing the novelty, soundness, and practical significance of the proposed framework. We respectfully hope the reviewers will consider updating their scores in light of these clarifications, and we thank the Area Chairs for their careful consideration of our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.