- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2(−1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates demographic bias in vision-language retrieval, noting that existing debiasing methods based on pre-trained vision-language models typically rely on text prompts to estimate bias direction while failing to fully utilize group distribution information from the actual retrieval gallery. To address this, the authors propose BALLAST (Barycentric Level-Allocated Latent Sparse Transport), a gallery-conditioned debiasing framework that requires neither model retraining nor per-image group annotations. The method first uses attribute prompts and a frozen CLIP model to estimate soft group memberships within the reference gallery; it then maps image features to a sparse representation space using a frozen sparse autoencoder and employs partial optimal transport based on Wasserstein barycenters to minimize feature distribution discrepancies across groups. Building on this, the authors introduce a closed-form query equalization method that constructs a bias subspace using group centroid vectors and eliminates the text query's average score bias toward specific groups via projection. Experimental results demonstrate that BALLAST significantly reduces KL divergence and MaxSkew under standard ViT retrieval settings and exhibits stability across various ablation studies and cross-query evaluations.
Critical strengths
- The research motivation is clear, and a practically significant perspective on gallery-conditioned debiasing is proposed.
- The method features a comprehensive mathematical formulation and a sound modular design.
- Experiments of this manuscript are extensive and yield impressive results.
Critical weaknesses
- This paper primarily employs KL divergence and MaxSkew—measuring the discrepancy between the demographic distribution of retrieval results and the overall gallery distribution—to evaluate debiasing effectiveness; however, reduced demographic distribution bias does not necessarily equate to superior retrieval quality. Experimental results indicate that current debiasing operations may compromise valid, query-relevant semantic information. Although the authors analyze this trade-off, the lack of relevance annotations for the primary stereotype-based retrieval tasks leaves it unclear whether fairness improvements are accompanied by broader semantic degradation in retrieval. Consequently, the paper needs to more fully explain how group fairness can be achieved without sacrificing retrieval relevance and should further discuss the task conditions under which these fairness objectives are applicable.
- Experimental results demonstrate that sparse transport itself is not the primary driver of performance gains. While additional transport operations yield consistent benefits in certain tasks, they entail complex steps such as frozen SAEs, sparse feature selection, quantile estimation, and distribution transport. Therefore, the paper needs to further articulate the unique advantages of these additional operations over simpler methods—such as gallery mean correction or dense feature space distribution adjustment—particularly when accounting for computational costs and potential losses in retrieval relevance.
- In real-world retrieval systems, gallery distributions may shift over time, across regions, or between data sources, rendering pre-estimated group centroids and distribution correction rules potentially obsolete. While the paper analyzes transferability across queries, query transfer and gallery distribution shift are distinct issues. Thus, it is recommended that the authors further evaluate the impact of reference gallery size, changes in group proportions, and cross-dataset distribution shifts on debiasing effectiveness, thereby more robustly substantiating the applicability of the gallery-conditioning approach in practical deployments.
Questions for authors
- It is recommended that the authors further evaluate the impact of reference gallery size, changes in group proportions, and cross-dataset distribution shifts on debiasing performance, so as to more robustly substantiate the applicability of the gallery-conditioning method in real-world deployment.
- Current experiments primarily utilize reference and evaluation galleries drawn from the same dataset; performance has not yet been validated for scenarios involving distribution shifts between the reference data and the actual retrieval gallery. Since both BALLAST’s transfer mapping and query projection subspace rely on statistics from the reference gallery, its generalization capability across varying population proportions and data distributions remains unclear.
- Could the authors provide additional experiments involving changes in group proportions or distribution shifts between the reference and target galleries, to verify whether the pre-computed transfer maps and query equalization remain effective?
- Given the high computational cost of the full BALLAST method, could the authors provide a further comparison of efficiency across different graph database sizes and query volumes, and explain under what circumstances the full method is preferable to BALLASTeq?
First Update After the Responses
I thank the author for their responses and their effort to address all concerns. What I do not appreciate is receiving super long responses and tables flooded with results and metrics, while avoiding to answer to the point. LLMs make this easy but I am not going to effectively play the game "can the human beat the machine?". Also, this practice is against the guidelines received by the PCs in a recent email "We highly encourage brevity: the review process is a significant amount of work for both reviewers and authors, and language models make it easy to generate long text that reviewers need to read, digest, and respond to."
I see that my ratings disagree with other reviewers and I am concerned to see if I have been unfair. I am processing the responses (to the extend that their length and dense and tough meanings allow me - LLM style responses) soon after they are posted to give the advantage of time to the authors. I am doing an genuine effort here and I would appreciate if the authors do the same in the follow up.
W1: Using the labels-conditional KL swaps the comparison to BaseCLIP, the baseline has better KL and actually even better precision according to K1. Variants eq and variant [email protected] improve that but these are not the main method and they are likely tweaked for this setting and single query. We do not know how this would perform in all other experiments, and additionally, big part of the contribution is left out in this variant.
W2: I agree with authors that it is the trade-off that matters. Wrt baseclip the proposed method trades a little of precision for KL. But SEM has much higher precision. We should see SEM spanning this trade-off? How is it possible to get more points, eg by different parametrization of SEM, on this trade-off? SEM can likely exchange bit part of its precision for some lower KL.
All in all, I see that in both my major weaknesses, there are still unresolved issues.