acceptodds
Under review as a conference paper at ICLR 2027

Distributed Geographic Identity Encoding in Vision-Language Models: From Linear Probing to Sparse Autoencoder Feature Ablation, and Why Post-Hoc Debiasing Fails

Abstract

Vision-Language Models (VLMs) are increasingly deployed on tasks — credit assessment, job-ad targeting, content moderation — where their representation of geographic identity has direct fairness consequences. Building on linear-probe recovery of macroeconomic signals from satellite imagery and street-view wealth estimation, we show that decoder-only VLMs encode country identity as a structural property of the residual stream. A ridge probe at Gemma-4 layer 6 recovers GDP PPP at 5-fold cross-validation R² = 0.33 but collapses under leave-one-country-out to R² = -0.05 (gap Δ = 0.38); this LOCO-collapse signature recurs across four architectures, with the direction emerging at layer 1 and not inherited from the vision tower. The subspace resists every post-hoc intervention we tested — input PGD, sparse-autoencoder feature ablation, universal adversarial perturbation (UAP), ROME weight edits, LoRA fine-tuning, and closed-form linear erasure (INLP, LEACE) — in a characteristic way each: geometric erasure leaves behavior intact (PGD: 84.2% geometric erasure, -7.4% behavioral shift; likewise ROME, LoRA, and closed-form LEACE), the feature dictionary offers no concentrated structure to ablate (SAE), and the only behavioral movers (UAP; INLP on one architecture, matched by a random control) act by collapsing predictions rather than removing the signal. The signal is nonetheless consequential: a single UAP direction gives Regress-To-Mean-specific gap reduction of 5.06× [3.61, 6.70], transfers cross-domain to FairFace face stimuli (3.24×; no shared linear wealth axis, r=-0.08), and closes the mortgage approval gap by 49.5%. We argue country-identity is structurally encoded, distributed, and behaviorally consequential; closing the output gap is not evidence the signal was removed, so post-hoc debiasing of VLMs is unreliable. We propose the Variable Dependence Spectrum for deployment-time triage and identify training-time concept regularization as the remaining intervention surface.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, interpretability, linear probing, sparse autoencoders, universal adversarial perturbations, concept erasure, representation geometry, fairness, geographic bias

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Q71COpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how Vision-Language Models (VLMs) internally represent "country identity" (geographic identity). The authors' central claim is that when a linear probe recovers macroeconomic signals such as GDP PPP from VLM hidden states with high accuracy, this does not reflect an encoding of the abstract concept of "wealth." Instead, it reflects a country-identity direction that has memorized which country is being seen. They detect this with Leave-One-Country-Out (LOCO) cross-validation, and claim this collapse signature replicates across four architectures. They further show that the signal is distributed across the residual stream and resists all six classes of post-hoc intervention tested (PGD, SAE feature ablation, UAP, ROME, LoRA, and closed-form linear erasure via INLP/LEACE). The signal is also behaviorally consequential: a single UAP direction reduces the mortgage approval gap, but this comes from collapsing predictions toward the unconditional mean (distributional collapse) rather than from removing the signal. From these results the authors argue that a reduction in the output gap is not evidence that the signal was removed, that post-hoc debiasing of VLMs is unreliable, and that the remaining intervention surface is training-time concept regularization.

Critical strengths
  • The paper challenges a widely shared assumption in interpretability research, namely that high linear-probe accuracy implies that the concept is understood. It poses the competing hypothesis that the probe may be reading nothing more than a memorized country identity, and it operationalizes this as a testable LOCO gap. LOCO itself has been used in prior work, but this paper brings it into the probing of VLM hidden states and moves the discussion from "demonstrating that bias exists" to "causally asking whether it can be removed." As a cautionary note on the interpretation of probing results, this has clear value.
  • Confirming both the LOCO collapse and the L1 birth across four families from different developers and with different architectures (Gemma-4/3, Qwen3-VL, MiniCPM-V) is a real merit. Observing the same tendency across different weights supports the claim that, at a minimum, these phenomena are not specific to a single model.
Critical weaknesses
  • The paper claims that the country-identity signal resists all six classes of post-hoc intervention. However, as the authors themselves acknowledge, there is no positive control showing that the same pipeline can successfully intervene on an attribute that is trivially manipulable. As a result, it is not possible to separate "the intervention failed because the signal is robust" from "the intervention failed because the pipeline is simply non-functional on this model class." The paper's strongest claim, and the conclusion drawn from it that "training-time intervention is the remaining path," rests entirely on the fact that all six interventions failed. Because the cause of that failure is undetermined, the falsifiability of the conclusion itself is undermined.
  • The paper makes strongly general and assertive claims such as "consistent across four models" and "training-time is the only remaining intervention surface," yet the design has several constraints: (a) five of the six interventions were run on Gemma-4 alone; (b) only one city per country was used, so a city-specific visual fingerprint cannot be separated from country identity; and (c) validation is concentrated at roughly the 4B scale. Point (b) bears directly on the central interpretation, since it cannot rule out the possibility that the LOCO collapse reflects the failure to generalize a single city's visual fingerprint rather than the failure to generalize a country's economic properties. There is a non-negligible gap between the strength of the conclusions and the reach of the evidence, which leaves the claims and the evidence insufficiently aligned.
Questions for authors

The following questions all bear on the assessment of the paper.

  1. For the five interventions run on Gemma-4 alone, is there evidence that the core geometry-behavior dissociation replicates on at least one additional architecture? Can the authors clarify precisely which claim the phrase "across four architectures" in the Abstract applies to, namely the LOCO diagnostic or the intervention resistance?

  2. Is the phrase "training-time concept regularization is the remaining intervention surface" intended to exclude other untested post-hoc methods (such as nonlinear erasure or architectural modification)? Or is it a qualified claim limited to "within the six classes tested in this paper"? Can the authors clarify the scope of this statement?

Reviewer_fVfrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether VLMs encode national wealth or only which country an image comes from. Ridge probes on hidden states predict GDP PPP from street-view images under random K-fold splits, but the fit collapses under leave-one-country-out (LOCO) validation in four ~4B VLMs, which the authors call a country-identity direction. They test six post-hoc interventions and report that none removes the concept behaviorally. They conclude that post-hoc debiasing of VLMs is unreliable and that output-parity audits can be misleading.

Critical strengths

LOCO as a probing check:
Using leave-one-country-out validation for VLM probes is a sensible import from poverty mapping, and the K-fold vs. LOCO gap is reported for four architectures.

Some well-chosen controls:
The matched-rank random-direction control that reproduces the INLP effect on Qwen3-VL is the kind of check erasure papers often skip.

A relevant audit question:
Asking whether a closed parity gap reflects removed signal or just shifted outputs matters for deployed systems, even if the paper does not answer it convincingly.

Critical weaknesses

The dataset is too small to support the main claim:
The study uses nine cities, one per country, and each country has a single GDP value. In the LOCO test, the probe must estimate a 9th GDP value after seeing only 8, so a poor result is expected regardless of whether the model encodes wealth. The finding therefore does not show that VLMs "memorize country identity rather than encode wealth." Several cities per country, or many more countries, would be needed to test this. Calling the effect "universal" across VLMs is also too strong, given that only four similarly sized models were tested.

The negative intervention results are hard to interpret:
The paper never shows that its intervention methods can change anything in these models, and the authors acknowledge this. Without that, "the methods failed" and "the concept cannot be removed" look the same. Some failures come from the setup rather than the models. The ROME edit was too small to register in bf16 precision, and LoRA was trained on a geometric objective rather than on the behavior it was meant to change.

The paper is hard to read and does not follow the required format:
Abstract and writing.
The abstract is a long list of numbers, CIs, and method names, without a clear statement of the problem and contribution, and should be much shorter. The abstract and introduction read as LLM-drafted, consistent with the AI use statement, with heavy bolding and many coined terms (defense gap, fairness laundering, VDS, birth layer).
Appendix placement.
The appendix comes before the references, but the format requires it after them. Reviewers are not required to read the appendix, yet the main text relies on it for key evidence, so several core claims are not supported in the main paper.
Figures.
Figure 1 uses most of a page to show two bars that the table below it already reports. Its numbers (0.346) also do not match the text (0.33), and the Figure 2 caption says "green" for bars that are blue.

Questions for authors

Can you show any attribute that your intervention pipeline successfully steers on the same models?

What is the gap reduction under random perturbations for the mortgage and ad prompts, and is the UAP effect significantly larger?

Reviewer_4UFDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether VLMs encode country-level economic information, such as GDP, in a way that generalizes across countries. It then asks whether that information can be removed after training. The authors probe four open VLMs on street-view images from nine countries. Probes that work under random cross-validation fail when a whole country is held out. The authors take this as evidence that the models encode country identity rather than wealth. They then apply a range of post-hoc interventions to the probe direction. These either leave behavior unchanged or shift it only by pushing predictions toward the mean. They conclude that post-hoc debiasing of VLMs is unreliable.

Critical strengths

The use of leave-one-country-out validation is the most useful part of the paper. Probing work in interpretability rarely tests whether a direction generalizes to held-out entities, and the paper makes a good case that it should. The probing result is repeated across several model families, and the authors are careful to report uncertainty and their own negative results. The breadth of the intervention study is also a strength, even though I have concerns about what it shows.

Critical weaknesses

My main concern is that the experimental design does not support the paper’s central conclusions.

The first issue is with the memorization claim. The economic labels are country-level, and each country contributes images from a single city. This leaves very few distinct label values. Under these conditions, failing a held-out-country test is close to expected even for a model that does carry some wealth signal. The design therefore cannot cleanly separate memorization from a weak or noisy representation. Labels that vary within a country would help. By the authors’ own taxonomy in the appendix, GDP also falls below the readability threshold at which they say their diagnostic applies.

The second issue is with the intervention study. The paper argues that the probe direction reflects memorization rather than the model’s use of wealth. It then treats the failure to change behavior by erasing that direction as evidence that the concept is distributed. These two arguments pull against each other. A direction the model does not rely on would not change behavior when removed, whatever the underlying encoding. The authors note that there is no positive control showing their pipeline can steer a simpler attribute. Without one, the null results are hard to interpret. In one case the pipeline’s reach is itself in doubt. On Qwen3-VL, adding large noise at the image inputs leaves the output unchanged, so attacks at that point cannot affect behavior regardless of how the concept is encoded.

The third issue is that several claims go beyond the evidence. The fairness results are presented as a strong headline, but the appendix shows the effect on the mortgage and ad tasks is not distinguishable from random perturbations. The conclusion about post-hoc debiasing is drawn from interventions that all target a probe direction or an input perturbation. Output-level debiasing methods are not tested. The models studied are all of similar size, which makes “universal” too strong.

On novelty, prior work has already shown that models encode geographic and socioeconomic information, and the paper acknowledges this. The contribution therefore rests on the held-out-country reinterpretation and the intervention study. For the reasons above, I do not think either currently supports the conclusions.

The paper is also hard to read. The main text is very dense with numbers and terminology. In places the appendix contradicts it, for example on which interventions were replicated across architectures.

Questions for authors
  1. How is the held-out-country R² computed when the held-out country has only one label value?
  2. Can you show that your intervention pipeline changes behavior for an attribute known to be steerable?
  3. Were the probe directions extracted under the same prompts used to measure behavior?
  4. Does the “born at layer 1” result still hold when you probe image-token positions at every layer, rather than comparing text tokens with image tokens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.