acceptodds
Under review as a conference paper at ICLR 2027

Condition on the Gallery, Not Just the Query: Query Equalization and Sparse Transport for Debiasing Vision-Language Retrieval

Abstract

Text-to-image retrieval with a vision-language model such as CLIP ranks the images of a gallery against a text query, and for a stereotyped query the retrieved set can have a demographic composition far from that of the gallery. A retrieval score has two arguments, the query and the image, yet many post-hoc corrections derive their edit from text prompts alone. We propose BALLAST (Barycentric Level-Allocated Latent Sparse Transport), a debiasing method for frozen CLIP models that conditions the correction on unlabeled images from the gallery's population. From prompts naming the protected attribute, BALLAST gives each reference image a soft membership over the attribute groups, and this one estimate drives two corrections: a repair of the gallery, which moves the groups' activations in the code of a frozen sparse autoencoder part of the way toward a shared barycenter on the coordinates where the groups differ, and a closed-form equalization of each query against the repaired gallery's group centroids. No per-image annotation is read and no gradient-based training takes place. On the SEM benchmark, with three retrieval datasets and four CLIP backbones, BALLAST lowers both KL divergence and MaxSkew below every SEM variant in all 24 comparisons on the two ViT backbones where it cuts KL on the stereotype queries by a factor of 2.3 to 5.0 relative to the best SEM variant per cell. The query equalization supplies most of the improvement, and its subspace has to come from the gallery: the same projection built from the attribute prompts alone raises KL above base CLIP on all 24 retrieval cells of the benchmark.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-language models, Fair retrieval, Optimal transport.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FFCxOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2(−1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates demographic bias in vision-language retrieval, noting that existing debiasing methods based on pre-trained vision-language models typically rely on text prompts to estimate bias direction while failing to fully utilize group distribution information from the actual retrieval gallery. To address this, the authors propose BALLAST (Barycentric Level-Allocated Latent Sparse Transport), a gallery-conditioned debiasing framework that requires neither model retraining nor per-image group annotations. The method first uses attribute prompts and a frozen CLIP model to estimate soft group memberships within the reference gallery; it then maps image features to a sparse representation space using a frozen sparse autoencoder and employs partial optimal transport based on Wasserstein barycenters to minimize feature distribution discrepancies across groups. Building on this, the authors introduce a closed-form query equalization method that constructs a bias subspace using group centroid vectors and eliminates the text query's average score bias toward specific groups via projection. Experimental results demonstrate that BALLAST significantly reduces KL divergence and MaxSkew under standard ViT retrieval settings and exhibits stability across various ablation studies and cross-query evaluations.

Critical strengths
  1. The research motivation is clear, and a practically significant perspective on gallery-conditioned debiasing is proposed.
  2. The method features a comprehensive mathematical formulation and a sound modular design.
  3. Experiments of this manuscript are extensive and yield impressive results.
Critical weaknesses
  1. This paper primarily employs KL divergence and MaxSkew—measuring the discrepancy between the demographic distribution of retrieval results and the overall gallery distribution—to evaluate debiasing effectiveness; however, reduced demographic distribution bias does not necessarily equate to superior retrieval quality. Experimental results indicate that current debiasing operations may compromise valid, query-relevant semantic information. Although the authors analyze this trade-off, the lack of relevance annotations for the primary stereotype-based retrieval tasks leaves it unclear whether fairness improvements are accompanied by broader semantic degradation in retrieval. Consequently, the paper needs to more fully explain how group fairness can be achieved without sacrificing retrieval relevance and should further discuss the task conditions under which these fairness objectives are applicable.
  2. Experimental results demonstrate that sparse transport itself is not the primary driver of performance gains. While additional transport operations yield consistent benefits in certain tasks, they entail complex steps such as frozen SAEs, sparse feature selection, quantile estimation, and distribution transport. Therefore, the paper needs to further articulate the unique advantages of these additional operations over simpler methods—such as gallery mean correction or dense feature space distribution adjustment—particularly when accounting for computational costs and potential losses in retrieval relevance.
  3. In real-world retrieval systems, gallery distributions may shift over time, across regions, or between data sources, rendering pre-estimated group centroids and distribution correction rules potentially obsolete. While the paper analyzes transferability across queries, query transfer and gallery distribution shift are distinct issues. Thus, it is recommended that the authors further evaluate the impact of reference gallery size, changes in group proportions, and cross-dataset distribution shifts on debiasing effectiveness, thereby more robustly substantiating the applicability of the gallery-conditioning approach in practical deployments.
Questions for authors
  1. It is recommended that the authors further evaluate the impact of reference gallery size, changes in group proportions, and cross-dataset distribution shifts on debiasing performance, so as to more robustly substantiate the applicability of the gallery-conditioning method in real-world deployment.
  2. Current experiments primarily utilize reference and evaluation galleries drawn from the same dataset; performance has not yet been validated for scenarios involving distribution shifts between the reference data and the actual retrieval gallery. Since both BALLAST’s transfer mapping and query projection subspace rely on statistics from the reference gallery, its generalization capability across varying population proportions and data distributions remains unclear.
  3. Could the authors provide additional experiments involving changes in group proportions or distribution shifts between the reference and target galleries, to verify whether the pre-computed transfer maps and query equalization remain effective?
  4. Given the high computational cost of the full BALLAST method, could the authors provide a further comparison of efficiency across different graph database sizes and query volumes, and explain under what circumstances the full method is preferable to BALLASTeq?
Reviewer_KAAeOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies text-to-image retrieval from the point of view of bias to protected attributes and fairness. The goal is for retrieval not to unfavorably retrieve particular groups more than others for some queries. The authors propose a method based on sparse auto encoders where they operare on SAE-atom strengths to unbias the representations. This paper actually builds on top of SEM-2026.

Critical strengths

Debiasing at the atom level is motivated. Aligning atom activation distributions of the groups is principled and keeps edits somehow interpretable

Debiasing retrieval with respect to protected-attribute groups is important, and postdoc methods rely on heuristics, while this paper takes a more principled approach.

The proposed approach is technically sound and computationally cheap to perform for a new query.

Critical weaknesses

W1 - a first major flaw in the evaluation The two metrics KL and MaxSkew are measuring how much the retrieval distribution over groups matches the prior of the dataset. But it should be done with respect to the prior over relevant images, and not all images. For example if gender balance is 60-40% (w/m) in the dataset, but for "a photo of a builder" the balance in the database is 30-70%, then one should be retrieving based on the latter and not the former. One cannot effectively perform this study without access to labels or at least some kind of pseudo-labels (which is tough to have them unbiases, so a chicken-egg problem). It is the labels-conditional KL that one needs.

W2 - a second major flaw in the evaluation - but related to the first one Good KL and MS the way the are computes says nothing by itself. A uniform number generator for similarity would retrieve the optimal score. The value is once someone sees this as a trade-off: debiasing vs retrieval quality. Table 3 which is the only one that reports precision with respect to semantic labels shows that the method sacrifices a big part of precision for debiasing. Therefore, I consider the method not meaningful. The point would be to observe better precision for same or better debiasing and/or better debiasing for better or same precision.

W3 - The protected “content” atoms come from “addict” and are reused for, say, “a photo of a doctor”, where they are irrelevant, while the atoms that matter for the new query get no protection. Why is this expected to generaize?

Post rebuttal: Given the acknowledgment of the value of the KL vs relevant examples, and its reporting alongside the original metric, the paper is at a better state than originally. This leads me to raise the score but up to weak rejection since I think the paper is not ready. The precision vs KL trade-off is showing that ballast does not stand well on this trade-off (both with respect to Table G5 and Table 41). Additionally, when there are no labels available I do not think the presented metrics are helpful and actually of much lower value than the ones on the datasets with labels. Reporting how much of the CLIP ranking did the method maintain is not helpful: the original ranking is the biased was the ones claimed to be biased in the first place.

Questions for authors

Weaknesses 1 and 2 are my main reason for rejection. Therefore I d expect the authors to respond to those and discuss whether there is an underlined misunderstanding of mine related to the task or evaluation.

1 comment
Reviewer_KAAeOpenReview ↗

First Update After the Responses

I thank the author for their responses and their effort to address all concerns. What I do not appreciate is receiving super long responses and tables flooded with results and metrics, while avoiding to answer to the point. LLMs make this easy but I am not going to effectively play the game "can the human beat the machine?". Also, this practice is against the guidelines received by the PCs in a recent email "We highly encourage brevity: the review process is a significant amount of work for both reviewers and authors, and language models make it easy to generate long text that reviewers need to read, digest, and respond to."

I see that my ratings disagree with other reviewers and I am concerned to see if I have been unfair. I am processing the responses (to the extend that their length and dense and tough meanings allow me - LLM style responses) soon after they are posted to give the advantage of time to the authors. I am doing an genuine effort here and I would appreciate if the authors do the same in the follow up.

W1: Using the labels-conditional KL swaps the comparison to BaseCLIP, the baseline has better KL and actually even better precision according to K1. Variants eq and variant [email protected] improve that but these are not the main method and they are likely tweaked for this setting and single query. We do not know how this would perform in all other experiments, and additionally, big part of the contribution is left out in this variant.

W2: I agree with authors that it is the trade-off that matters. Wrt baseclip the proposed method trades a little of precision for KL. But SEM has much higher precision. We should see SEM spanning this trade-off? How is it possible to get more points, eg by different parametrization of SEM, on this trade-off? SEM can likely exchange bit part of its precision for some lower KL.

All in all, I see that in both my major weaknesses, there are still unresolved issues.

Reviewer_ESCGOpenReview ↗
Rating
1(−2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propos "BALLAST (Barycentric Level-Allocated Latent Sparse Transport)", using a batch of unlabelled images from the same population as gallery and uses prompts such as "a photo of a man/woman/black person" etc. to guess how much each image seems to belong to each group. These are the so called soft groups. Then, it uses a spare auto encoder, it moves the feature distributions partway toward a shared average target in case of pseudo-groups that use certain features differently. It repairs the gallery by adjusting the query so on average it does not score a group higher than another using closed-form projection instead of classical gradient training.

The main messages of the paper are that this method works well on stereotype retrieval benchmarks (especially VIT CLIP models), and that projection must be built form actual gallery images, not just text prompts. Using only text prompts can even make it worse.

Critical strengths
  • Great central idea: debasing retrieval with respect to actual gallery is very important
  • label-free, simple method - great real-world transfer possible
  • broad experimental evaluations
Critical weaknesses
  • instead of only using CLIP (that was not trained on a public dataset), you could include OpenCLIP or a second opaque dataset model such as SigLIP from google, so you could have a more realistic baseline to compare to with more recent or open architectures
  • even though the authors include a fixed-configuration experiment, this creates a real risk that the method is partly tuned to the evaluation protocol rather than to robust, generalisable fairness behaviour. The evaluation would be much stronger if it included independent relevance orutility metrics across all tasks, human semantic validation or external validation not sharing the same proxy metric used for calibration..
  • entire methodology relies on noisy soft pseudo-groups and a caption-trained SAE applied to images, both of which have significant limitations

Minor language issues:

  • equalization´s instead of equalization (Table 2 legend)
  • maybe state what DIAL stands for - i didn't know the acronym and the title of your cited paper also does not disclose it - only a quick reference check and abstract reading discloses the acronym: Disentangle, Identify, And Label-free removal

Maybe add a bigger ethical section - you started nicely in the introduction - where you discuss the following "bias" theme:

Bias is a difficult topic, both in datasets and models. Datasets contain "positive" and "negative" bias at the same time, and this has to be more clearly assessed, at least in the appendix section. Sometimes you want to have a "real-world reflection" of the data (e.g., you want to find out how many female birds are in South America, and logically, you have a bias towards female or male birds if there is a discrepancy - but you really want to know exactly that bias! And there is a negative bias: if two features are correlated and you train a predictor, and you want a neutral assessment, this "positive" dataset bias (in the above sense that reflects the "perception" of the internet, the real world etc.) becomes "negative", as it discriminates against a person if a bad features is linked to a "neutral" transport medium for "mitigator" reasons, e.g., preference in the internet datasets, or marvellous intentions of dataset creators.

Questions for authors

I have some general questions (that might already be answered in the paper in parts that I do not fully understand):

  • How sensitive is BALLAST to mismatch between the calibration reference gallery and the deployed gallery, or to a bad seed query family??
  • What can you say about failure modes when the soft pseudo-groups are wrong, especially for race (e.g., if your "human" annotation tests showed that "caucasian-white man" is actually a different one?
  • Would it be possible to give a principled fairness-utility operating curve that does not rely solely on optimising the same metric family used for the evaluation?
  • You state "Code, configurations, and per-configuration records are released." - what License?
4 comments

Dear Reviewer, I would appreciate if you could also comment on my weaknesses W1 and W2. I think these are major issues arguging for rejection and the responses do not yet address them.

Dear reviewer KAAe,

you raise a valid point - I agree with your general critical assessment of WP1/ WP2: KL and MaxSkew are proxy fairness metrics and are most meaningful when paired with the retrieval utility (ideally under a relevance-conditional group prior when labels exist... when they exist!).

For the stereotype retrieval benchmark used here, such relevance labels are not available for the protected groups, so consequentially / logically, gallery / dataset prior is the standard CLIP-fairness proxy. The authors have now also reported label-conditional KL and precision on CelebA attributes, where BALLAST or BALLAST_eq improve fairness while preserving utility.

Their added precision / trade-off, fixed-configuration, seed-query, and attribute-utility analyses etc. therefore address the substance of of WP1 and 2 in the (!) measurable cases, while revealing the remaining limitation of prompt / group sensitivity and proxy-dependent evaluation.

I would thus treat W1/W2 as important caveats that strengthen the paper - after all, to me it is more a limitation / evaluation-protocol point issue instaed of a fatal flaw leading to clear rejection. But still I agree with your general stance, and this limitation is another reason for me not to raise my score.

Dear ESCG, thank you for taking a look and commenting.

As per my post a few minutes ago, I do not think that in the measurable cases/datasets there is an achievement. On the contrary, SEM behaves much better. Let me know if I m seeing it wrongly please.

Now, for the cases/datasets where there is no labels, there, it seems we disagree. I do not think there is anthing meaningful to measure there. And the fact that prior work did it, does not necessarily mean we have to accept it and keep accepting papers that rely on it.

If you use KL vs gallery prior as a metric, then you are rewarding a method if it stopped capturing the concept/class you care and query about. Random ranking is the best!

Dear KAAe,

thanks for your clarifications.. On hair you are reading the measurable cases correctly, where base CLIP and SEM beat BALLAST on both the label-conditional KL and precision.

On the five gender-independent CelebA attributes, however, the authors report that on ViT-B/16 BALLAST is below SEM_i on that metric at higher precision, while SEM_b/bi win only on precision at about three times the KL, so I would not say SEM is generally better across all measurable cases.

On W1, I think our disagreement touches the same issue as the ethics comment (see my intiial review please). some queries call for a faithful snapshot of what is relevant, the "positive bias" that should be preserved, so a search for blond should return about 94% women, and your label-conditional KL is the right target. Others call for a deliberately balanced selection, for example a recruiter search where gender should play no roleand there gallery parity is the right target... Your chicken-egg problem arises for the first goal, which needs relevance labels, but the second needs only the group annotations the benchmarks already provide. :)

For stereotype queries like criminal there is no relevant set at all, and any estimate of one would encode the stereotype...

I therefore read the hair results as the cost of the authors' chosen goal rather than as evidence that the evaluation is meaningless (your clear rejection evaluation). The limitation is that the method applies parity to every query and relies on the operator to decide where gender legitimately matters, which the paper must state clearly!

On W2, I agree that utility has to be measured alongside KL... and that a scaled version of SEM edit would have made the trade-off comparison fairer in the first place.

Please correct me if i'm wrong.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.