When Geographic Priors Become Bias: Causal Steering in Remote-Sensing VLMs

Geographic context can be a useful prior in remote sensing, but it becomes problematic when it overrides fixed visual evidence. We study this transition from geographic prior to geographic bias in remote-sensing vision-language models (VLMs) using pixel-identical geographic counterfactuals, where only the stated location is changed while the visual input remains unchanged. This isolates geographic influence from genuine differences in landscape, infrastructure, or physical damage. Across socioeconomic assessment and disaster-damage estimation, we find that geographic cues can substantially shift predictions even when the underlying imagery is identical, with external RWI and damage annotations revealing when such shifts are unsupported. We then use activation patching to identify transformer representations that causally mediate these effects and construct bias-weighted steering directions for inference-time intervention. Steering consistently reduces geographic counterfactual gaps on unseen images while preserving task-relevant utility. To the best of our knowledge, this is the first framework to jointly audit, causally localize, and selectively steer geographic bias in remote-sensing VLMs under controlled pixel-identical counterfactuals. Our results show that geographic priors can become causal shortcuts, but that these effects can be systematically identified and mitigated without retraining.