acceptodds
Desk rejected at ICLR 2027

FBCIR: Diagnosing and Mitigating Partial-Query Reliance in Composed Image Retrieval

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Composed image retrieval (CIR) is inherently conjunctive: a correct target must satisfy both a reference image and a textual modification. Yet a model can often retrieve the correct target without using the full composed query, because conventional easy negatives can be rejected from only one query component. We refer to this behavior as focus imbalances. We propose FBCIR, a multi-modal focus interpretation method that identifies visual and textual input components sufficient to preserve a model's retrieval decisions. Motivated by this diagnosis, we further develop FBCIR-Data, a CIR data augmentation workflow that enriches existing datasets with curated hard negatives requiring discrimination of both visual and textual constraints, together with a controlled benchmark and finetuning data for evaluating and mitigating focus imbalances. Experiments across representative CIR models reveal substantial vulnerabilities under such hard-negative settings. Under matched finetuning configurations, FBCIR-Data consistently improves hard-case retrieval for VLM-based retrievers and outperforms alternative training datasets, with additional gains observed on standard benchmarks. These results show that explicitly exposing models to negatives that invalidate single-component shortcuts provides an effective way to diagnose and improve robustness in composed image retrieval.

Primary area:
datasets and benchmarks
Keywords:
composed image retrieval, interpretable AI, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Xo7KOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the issue of unimodal dependency in Composed Image Retrieval (CIR), noting that existing retrieval models may rely on local information from either the image or the text alone, failing to fully integrate the bimodal semantics of the composed query. To analyze this phenomenon, the authors propose FBCIR, a multimodal attention diagnostic method based on iterative token pruning; it quantifies the model's dependency on different modalities by identifying the image regions and text tokens that preserve the original retrieval ranking. Building on this, the authors introduce the FBCIR-Data augmentation framework, which leverages vision-language models, image editing models, and image generation models to construct targeted hard negative samples, creating corresponding evaluation benchmarks and fine-tuning datasets. Experiments cover various CLIP-based and VLM-based retrieval models, evaluating the proposed method on benchmarks such as FBCIR-Data, CIRR, FashionIQ, and GeneCIS. Results indicate that the constructed hard negative samples improve the fine-grained retrieval capabilities of certain models and mitigate the defined modality attention imbalance under several experimental configurations. However, the paper faces several key issues. First, the proposed attention imbalance metric relies heavily on heuristic token retention ratios, and there is insufficient validation regarding whether it accurately reflects the model's true cross-modal dependency. Second, improvements in retrieval performance do not always align with reductions in attention imbalance, and since the diagnostic module does not directly guide the data augmentation process, it cannot be conclusively proven that the performance gains stem from the proposed attention-balancing mechanism. Furthermore, the semantic reliability of the synthetic positive and negative samples, as well as the method's generalization capability in independent hard retrieval scenarios, require further validation.

Critical strengths
  1. This paper addresses the issue where compositional image retrieval models may rely on single-modality information to perform retrieval, investigating the problem through the lenses of model behavior diagnosis and data augmentation. This issue is closely linked to the fundamental requirement of compositional image retrieval (CIR) to satisfy joint constraints from both image and text, and understanding it sheds light on the failure modes of existing retrieval models when dealing with challenging candidate sets.
  2. The authors designed "hard negative" samples—specifically, samples that satisfy partial image constraints while violating text modification requirements, and others that satisfy partial text semantics while violating image constraints—to reduce, at the data level, the likelihood of models relying on single-modality information for retrieval.
  3. The paper conducts experiments on various CLIP-based and VLM-based retrieval models and compares the proposed approach with existing data augmentation methods. Additionally, the authors analyze the impact of factors such as training data scale, the proportion of hard negative samples, LoRA configurations, and the choice of image generation models, while also performing a manual assessment of data quality.
Critical weaknesses
  1. The core research motivation of this paper—namely, that existing composed image retrieval models may rely on single-modality shortcuts, thereby preventing standard benchmarks from adequately testing genuine multimodal composition capabilities—has already been systematically discussed in recent studies. In particular, Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (2026) revealed similar issues through direct single-modality interventions and manual verification, while i-CIR (NeurIPS 2025) and MA-CIR (ICCV 2025) have introduced modality-specific hard negative samples highly similar to those in the present work. Although FBCIR proposes a new token-pruning diagnostic metric and a generative data augmentation pipeline, the paper fails to sufficiently articulate its distinct contributions relative to these existing works; furthermore, it lacks direct comparisons to demonstrate that its complex diagnosis of attention imbalance is more effective than established single-modality intervention methods.
  2. This paper quantifies the level of attention paid to each modality by calculating the ratio of retained image regions versus retained text tokens, using the absolute difference between these two ratios as an indicator of attention imbalance. However, the ratio of image regions and the ratio of text tokens do not inherently possess comparable semantic significance; identical retention ratios do not imply that the model relies equally on both modalities for its decisions. Furthermore, zero-masking of image regions and the deletion of text tokens introduce distinct forms of input perturbation, the effects of which may be confounded with the true degree of modality reliance. Although the authors provide supplementary validation in Appendix C using token weights generated by a VLM, that experiment primarily demonstrates numerical consistency across different weighting schemes; it does not directly prove that the metric accurately characterizes cross-modal information utilization. Consequently, the paper's core conclusions regarding attention balance and its improvement lack a sufficiently reliable basis for measurement.
  3. The paper treats the final set of retained image regions and text tokens as the minimal sufficient set supporting the retrieval decision; however, the actual search employs a beam search with limited width, retaining only a finite number of valid states at each step. This approximate search strategy guarantees neither coverage of all possible token combinations nor that the final result constitutes a globally minimal sufficient set. Moreover, the impact of token deletion on retrieval ranking is not necessarily monotonic; a state deemed invalid during the progressive deletion process cannot be ruled out as potentially regaining its original ranking after the subsequent deletion of other tokens.
  4. While the paper frames attention diagnosis and data augmentation as two core components of a unified framework, the described augmentation process relies primarily on predefined rules for constructing hard negative samples; it does not dynamically determine negative sample types, sample selection, or training weights based on the diagnostic results from FBCIR. According to Figure 2, the diagnostic module functions mainly as an indicator of issues and a tool for post-hoc verification, rather than as an essential component of the data augmentation process itself. Therefore, the current experiments fail to demonstrate that the diagnostic module contributes meaningfully to the reported improvements in retrieval performance.
  5. Hard negative samples are constructed using image editing and generative models, while original positive samples in certain data sources are replaced with synthetic images, thereby repurposing the original positive samples as candidate negative samples. However, the fact that the original positive sample matches the query less closely than the generated target does not necessarily mean it fails to satisfy the composite query conditions; particularly in retrieval scenarios where multiple plausible targets may exist, such re-labeling could introduce false negatives.
Questions for authors
  1. Recent studies have systematically explored the unimodal shortcut problem and the construction of hard negative samples in composed image retrieval. For instance, Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (2026) revealed—through unimodal interventions and manual verification—that existing benchmarks inadequately assess true multimodal composition capabilities, while Instance-Level Composed Image Retrieval (NeurIPS 2025) and MA-CIR: A Multimodal Arithmetic Benchmark for Composed Image Retrieval (ICCV 2025) proposed targeted hard negative samples and evaluation schemes. Given the significant overlap in research motivation and methodological design between these works and the present paper, the authors are requested to further clarify the substantive innovations of FBCIR relative to the aforementioned studies; specifically, what unique contributions do the token-pruning-based attention imbalance diagnosis method and the generative hard negative sample construction strategy offer? Furthermore, could the authors provide more direct comparative experiments to validate the independent value and necessity of the proposed method compared to existing approaches for unimodal diagnosis and hard negative sample construction?
  2. Could the authors further explain why the retention ratio of image regions and the retention ratio of text tokens can serve as directly comparable metrics for modality dependency?
  3. When employing a finite beam size, how do the authors guarantee that the iterative pruning process ultimately yields the minimal sufficient set of tokens? In cases where token removal does not satisfy the monotonicity property, is it possible for the current search to overlook a smaller valid subset?
  4. FBCIR-Data primarily employs predefined methods for constructing hard negative samples and does not significantly rely on FBCIR diagnostic results. Could the authors provide experiments that directly utilize diagnostic results for sample selection, negative sample type assignment, or training weight adjustment—and compare these against standard hard negative sample augmentation that does not use diagnostic information—to demonstrate the actual connection between the two modules?
  5. In Table 3 and Appendix G.4, there are instances where retrieval performance improves significantly while Focus Imbalance remains virtually unchanged or even worsens. How do the authors explain these phenomena?
Reviewer_5uXJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates focus imbalance in composed image retrieval, where models may rely on only part of a multimodal query. It proposes FBCIR, a perturbation-based method for identifying influential image and text components, and FBCIR-Data, a benchmark and training dataset with targeted hard negatives. Experiments across multiple CIR models demonstrate improved hard-case retrieval and reduced focus imbalance after fine-tuning.

Critical strengths
  1. The paper investigates an important but underexplored issue in composed image retrieval, where models may exploit single-modality shortcuts.
  2. FBCIR-Data constructs targeted hard negatives that require models to consider both visual and textual constraints.
  3. The experiments cover multiple retrieval architectures, training datasets, and data scales, demonstrating consistent improvements in hard-case retrieval.
Critical weaknesses
  1. The focus imbalance metric relies on heuristic token weights, and equal modality contributions do not necessarily indicate better multimodal reasoning.
  2. The performance improvements may primarily result from hard-negative training rather than improved focus balancing. The contributions of different data augmentation components are not sufficiently isolated.
  3. The proposed benchmark contains limited queries with five candidates per query, limiting its coverage of realistic large-scale retrieval scenarios.
Questions for authors
  1. How does the proposed metric distinguish harmful modality shortcuts from legitimate differences in modality importance?
  2. Can the authors clarify the individual contributions of synthetic positives and different hard-negative types to the performance improvements?
Reviewer_mP9fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses partial-query reliance in CIR, where models may retrieve correct targets using only one query modality. The paper proposes FBCIR, an iterative pruning method that identifies image regions and text tokens sufficient to preserve retrieval rankings, and introduces a focus balance ratio to quantify modality imbalance. The paper further proposes FBCIR-Data, which augments existing CIR triplets with synthetic hard negatives and selectively replaces original positives. Experiments across several CIR models report improvements in hard-case retrieval, particularly for VLM-based retrievers, with additional gains on standard benchmarks.

Critical strengths
  1. The paper approaches CIR from an interesting modality-balancing perspective. Identifying single-modality shortcuts and improving data construction accordingly offers a useful direction beyond optimizing aggregate retrieval recall.

  2. The paper is clearly written and easy to follow, with well-organized figures and tables.

  3. The experiments demonstrate that FBCIR-Data can improve retrieval performance. Comparisons under matched fine-tuning settings support the practical value of the constructed data.

Critical weaknesses
  1. The negative-labeling protocol imposes insufficiently justified restrictions on valid CIR targets. Visual discrepancy or lower relevance than a synthetic positive does not establish query incompatibility, potentially introducing false negatives into both evaluation and training.

  2. The Focus Balance Ratio lacks sufficient semantic justification. Retained image-area and word-retention proportions are not calibrated measures of comparable semantic contributions, and numerical balance does not necessarily indicate better multimodal reasoning.

  3. The training objective is not tailored to the distinct roles of text-augmented, image-augmented, and identity negatives. Differential weighting of in-batch and in-sample negatives does not explicitly exploit the complementary supervision these categories are intended to provide.

  4. The ablations do not isolate individual negative types or positive replacement. Their respective contributions, and the role of focus balancing beyond generic hard-negative training, remain unclear.

Questions for authors
  1. Given prior work on object-level and instance-preserving retrieval settings [1–3], what explicit target-validity criteria distinguish FBCIR from standard CIR? How are generated negatives and relabeled original positives verified to violate these criteria?

  2. Have the authors considered type-specific losses or weighting schemes to better exploit the complementary supervision provided by the different negative categories?

  3. Could the authors provide component-wise ablations on the negative categories and positive replacement to clarify their individual contributions and the relationship between retrieval gains and focus balancing?

References:

[1] Composed Object Retrieval: Object-level Retrieval via Composed Expressions.

[2] Instance-Level Composed Image Retrieval.

[3] Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.