acceptodds
Under review as a conference paper at ICLR 2027

Distributed Local Evidence Allocation under Strong Global Anchors for Composed Image Retrieval

Abstract

Composed Image Retrieval (CIR) retrieves a target image from a reference image and a modification text. Recent supervised CIR methods complement strong vision-language pretrained (VLP) representations with specialized local modeling, but it remains unclear whether the benefit of local modeling under strong global representations is tied to such specialized local modeling pipelines. Through controlled studies, we find that global VLP features provide a strong retrieval anchor and that, in a controlled ENCODER study, replacing its specialized local branch with lightweight token compression causes only a small degradation. On FashionIQ, dense and compressed local representations also perform similarly once global-local branch weighting is controlled. These observations motivate a narrower design question: how should a compact local branch allocate its limited capacity across reference evidence for each modification? We propose DART, a Distributed Information Bottleneck (DIB)-inspired module that conditions token-wise allocation on modification semantics and regularizes the allocation coefficients around a compression prior. DART is combined with balanced global-local aggregation while preserving the host VLP backbone and multimodal composition mechanism. Experiments on FashionIQ and CIRR support distributed local evidence allocation as a compact local modeling strategy under strong global representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Composed Image Retrieval, Local Representation Learning, Multimodal Retrieval, Token Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GK2GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The task studied in this paper is combined image retrieval, which involves retrieving the target image from the candidate images based on the reference image and the modified text. Recent supervised CIR methods usually further incorporate specialized local modeling on top of strong visual-language pre-training (VLP) representations to complement the information provided by the global features. However, it is still unclear whether the benefits of local modeling depend on the specially designed local modeling process when strong global representations are already available. The authors first conducted a controlled analysis on the ENCODER and found that replacing its specialized local branch with lightweight token compression resulted in only a slight performance decline. At the same time, after controlling the global-local branch weighting on FashionIQ, the performance of dense and compressed local representations was relatively close. Based on these observations, the authors proposed DART, which allocates the limited compact local capacity to different reference image evidence according to the semantics of the modified text. The authors evaluated this method on FashionIQ and CIRR, covering three supervised CIR frameworks: ENCODER, COMBINER, and OFFSET, and achieved corresponding performance improvements overall.

Critical strengths
  1. This paper conducts a controlled experiment to analyze the individual roles of strong global VLP representations and compact local modeling. It does not make any direct assumptions, thus establishing the method's motivation on a clearer empirical basis.
  2. This paper proposes the DART module, which allocates reference-token weights token-wise based on modification semantics, enabling the compact local representation to adapt to the current modified text instead of performing fixed compression on all reference tokens. And The paper evaluates on three representative supervised CIR frameworks, namely ENCODER, COMBINER, and OFFSET, and covers FashionIQ and CIRR, demonstrating a certain degree of framework compatibility.
Critical weaknesses
  1. This paper only conducted a sensitivity analysis on the regularization coefficient β. However, there was a lack of corresponding parameter sensitivity experiments for the equally important hyperparameter compact-token count K. Moreover, the paper only studied β in the ENCODER framework and the FashionIQ dataset, lacking experiments in other frameworks and datasets, and failing to conduct sufficient parameter sensitivity experiments. Additionally, it did not provide corresponding Case Studies or baseline comparison retrieval success/failure cases, making it impossible to intuitively demonstrate from which types of text modifications the performance improvement of DART mainly comes.
  2. According to the description in the paper and the experimental presentation, DART seems more like a general enhancement plugin. Currently, the paper only integrates it into three supervised CIR frameworks: COMBINER, ENCODER, and OFFSET, ignoring other local modeling structure methods based on CLIP, and conducting experiments only on the FashionIQ and CIRR datasets, also neglecting the Shoes dataset in COMBINER, ENCODER, and OFFSET, which affects the applicability of the proposed DART module.
Questions for authors
  1. Besides β, can the author also conduct sensitivity experiments on K and carry out corresponding experimental analysis?
  2. Can the author provide the original framework and the successful and failed cases of DART's search in terms of retrieval?
  3. Do the current three representative frameworks and two datasets represent the main claims of the author regarding the generalization of the method? If the author intends to use DART as a more generalized plug-in module, can it be further verified on other local modeling frameworks based on CLIP and related benchmarks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GK2G: K Sensitivity, Qualitative Results

We thank the reviewer for the detailed and constructive feedback on hyperparameter sensitivity, qualitative analysis, and the applicability of DART across frameworks and datasets. All revisions are highlighted in blue in the revised manuscript.


Q1: Extended Hyperparameter Sensitivity Analysis

We thank the reviewer for highlighting the need for a more comprehensive hyperparameter sensitivity analysis.

Sensitivity to the Number of Local Tokens KK: We have included a KK-sensitivity analysis on FashionIQ within the ENCODER framework (Appendix A.2, Figure 6 (b)), comparing ENCODER, TL+BA, and DART across K=2,4,6,8K={2,4,6,8}.

The results show that the original ENCODER exhibits a substantial performance drop at larger values of KK, whereas TL+BA and DART remain relatively stable.

DART consistently outperforms TL+BA across the tested capacities, suggesting that modification-conditioned allocation provides benefits under different local token budgets.

Additional β\beta Sensitivity Analysis. We apologize that time constraints have prevented us from completing the β\beta sensitivity analysis across other frameworks or datasets. We will complete the analysis on FashionIQ with other frameworks during the remaining discussion period.


Q2: Qualitative Retrieval Results

Thank you for suggesting a qualitative analysis to better understand the types of queries where DART performs well and where it remains limited.

We have included qualitative retrieval comparisons between the original ENCODER framework and DART, including both successful and failed cases on FashionIQ (Section 5.4, Figure 5) , CIRR (Appendix A.3, Figure 7) and i-CIR (Appendix A.3, Figure 8).

The successful cases show that DART better satisfies modification requirements while retaining relevant information from the reference image. The failed cases reveal a remaining limitation in fine-grained discrimination of visual attributes, such as the degree of shininess, fur darkness, and illumination conditions.

AuthorsOpenReview ↗

Response to Reviewer GK2G: Expanded Dataset Evaluation

Q3: Generalizability Across Datasets and Frameworks

Thank you for raising concerns about the generalizability of DART.

Additional Evaluation on Shoes. We have included experiments on Shoes using all three frameworks (Appendix A.1, Table 6). The performance changes after introducing DART are summarized below.

FrameworkΔR@1ΔR@10ΔR@50ΔAvg. Recall
COMBINER−0.87−0.19−0.02−0.36
ENCODER+2.11+2.28+1.56+1.99
OFFSET+3.04+2.08+0.32+1.82

Δ denotes the performance change after adding DART to each reproduced baseline, calculated from the mean results over three independent runs. All values are in percentage points.

DART improves retrieval performance for ENCODER and OFFSET on Shoes but slightly reduces COMBINER’s performance. These results provide further evidence of DART's applicability across datasets, while highlighting that its benefits vary across host frameworks.

Additional Cross-Dataset Evaluation on i-CIR and CIRCO. We have additionally evaluated CIRR-trained ENCODER and COMBINER models and their DART variants on i-CIR and CIRCO without further fine-tuning (Appendix A.1, Tables 7 and 8).

As shown below, DART improves all reported metrics for both frameworks on i-CIR, as well as all reported mAP metrics on the CIRCO validation and test sets, with larger gains for ENCODER on both datasets. These results provide additional evidence of DART’s applicability beyond the training dataset.

FrameworkmAPmmAPminmAP
ENCODER37.8035.8212.62
ENCODER + DART39.9638.3613.76
COMBINER19.7920.135.47
COMBINER + DART20.6120.695.72

Cross-dataset retrieval results on i-CIR (%). All models are trained on CIRR and evaluated without additional fine-tuning.

ValidationTest
FrameworkmAP@5mAP@10mAP@25mAP@50mAP@5mAP@10mAP@25mAP@50
ENCODER22.2823.3025.2226.1321.9122.4024.4225.32
ENCODER + DART25.3426.1328.1929.1323.9624.5326.6727.58
COMBINER21.2622.0523.9524.8020.2920.8722.6323.55
COMBINER + DART22.3623.0324.8625.7020.4220.8922.7323.58

Cross-dataset retrieval results on CIRCO (%). All models are trained on CIRR and evaluated without additional fine-tuning.

Evaluation on Additional Host Frameworks. We apologize that the adaptation and baseline reproduction required for a new host framework could not be completed within the revision period. Our conclusions remain limited to the three evaluated frameworks.

AuthorsOpenReview ↗

Response to Reviewer GK2G: Additional Beta-Sensitivity Analysis

Q1 - 2: Extended Hyperparameter Sensitivity Analysis

Following our previous response, we extend the β\beta-sensitivity analysis to COMBINER and OFFSET on FashionIQ (Appendix A.2, Figure 6(a)).

ENCODER and COMBINER achieve their best results at β=0.5\beta=0.5, while OFFSET performs best at β=0.75\beta=0.75. DART exhibits relatively stable performance across the tested values in all three frameworks, supporting the use of β=0.5\beta=0.5 as a common setting.

We update Figure 6(a) and the corresponding analysis in the revised manuscript. Thank you again for your constructive suggestion.

Reviewer_US3COpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about composed image retrieval with training a model that combines textual and visual input via training on triplets (ref image, text modified, target image). The contribution is about how to aggregate mutiple local/patch tokens and the global token into a single final repreentation, within the model that also combines the two modalities. Evaluation is performed on two of the known datasets, CIRR and FashionIQ.

Critical strengths

The paper reveals that the existing models, even though are seeing local representation in the input, they mostly rely on the global features. This is the early experiment in fig 1 which paves the way for new architectures that better exploit local representation. Indeed to handle general composed queries one onften needs to transform the appearance of object parts (eg a building during the night, or an object folded, ...), therefore the better use of local features should be a key.

The proposed approach is shown applicable and effective on top of 3 existing models. Indeed its design is general, and the experiments support the general applicability.

Critical weaknesses

W1 - Reproduced scores are lower, on CIRR significantly, than the reported scores. It is not clear to me why, but it looks like 63.26 vs 62.23 Fashion/combiner, 81.92 vs 74.40 CIRCC/combiner, 62.59 vs 62.18 on Fashion/offset, and 91.99 vs 73.19 on CIRR/offset. Therefore, the shown improvements are either non significant or non existing.

W2 - The loss in (8) is not intuitive. Rho is mentioned as a compression factor (how many less local tokens there are), that is fine. But the way that rho_i is used in the model has nothing to do with compression even though it is referred to as compression. Rho_i acts as a temperature in modifying original attention weights. Therefore, a KL loss between one value and the other, and calling them both compression factors is not intuitive and seems wrong.

W3 - I know that both CIRR and FashionIQ have been established datasets and there have been tens of papers evaluating on them, but they are a distraction. There are a few reasons for that and I will explain. I have personally gone through tons of examples in those and in other CIR datasets.

  • For a large number of queries, the text solely does all the work, there is no composition required. You can clearly understand this qualitatively (a photo of a soft drink and "show three bottles of soft drink", the positives are the photos with 3 bottles of soft drinks)
  • There is a large amount of noise in the ground truth, in terms of false negatives. The labels include one or few positives, while all the rest are not labeled.
  • For most of the queries what you need from the image is to recognise the main object catogory, once you do this, then you don t need anything else from the image.
  • The above point has to do with the class definition, which is at the semantic-category level. The two photos, query and positive need to show an object of the same semantic-category, which is the object id does not matter or spatial layout or anything else about specific appearance of the image. These datasets do not reflect a real-world setting (toy datasets to help progress would be ok, but all the above issues made them just bad datasets), while there have been 2 datasets out there that focus on the correct direction. This is, we need to retrieve objects of the same identity (the same particular object) under different conditions. [a] Psomas et al, neurips 2025, i-CIR: Instance-Level Composed Image Retrieval [b] Rosasco CVPR 2024 ConCon-Chi: Concept-Context Chimera Benchmark for Personalized Vision-Language Tasks. Both [a] and [b] can really drive progess in CIR while CIRR and FashionIQ have been providing a very noisy measurement.

W4 - The novelty is kind of limited, a FiLM like text conditioning, but ok, the message is on the learning how to aggregate local tokens. In that sense, it would be nice to compare with other aggregators that are straight-forward to apply, eg cross-attention pooling, plain soft-max.

W5 - the writing is not self-contained (eg background material from the existing models) and the paper is written in a way that it requires two rounds of reading (technical terms used before being explained). A particular issue that is major is thaty the architecture of F_theta is never explained, despite being central in the method.

After rebuttal

  • W1: It seems the authors work on some settings where prior work does not achieve its best results. Also, even if the authors use original settings for some of the prior methods, reproduction is not achieved. Therefore, it is unclear if improvements hold in the good performance regime of prior work, and also unclear why the authors insist to evaluate only on those.
  • W2: The rebuttal restates intent instead of justifying the design.
  • W3: The response is satisfactory in the sense that the authors acknowledge the dataset limitations and include the suggested ones. Their method appears to improve. I would also though prefer a final paper to present more of the experiments and ablations on those, rather than the original ones with their issues

As a consquence, I choose to maintain the original score

Questions for authors

Both W1 and W3, each by itself, are reasons to reject the paper. The authors should clarify about W1 and also comment on W3. The fact that a lot of papers have followed that path, to me, does not mean that we need to continue accepting papers that do it wrongly, when there are alternative datasets or possibilities to create new ones.

W2 requires the authors reply too, while W4 can be handled by some extra experiments.

8 comments
AuthorsOpenReview ↗

Response to Reviewer US3C (W1 - 1): Baseline Reproduction

We thank the reviewer for the detailed and constructive feedback.

In response, we clarify the reproduction settings and regularization design, provide additional evaluations on i-CIR and CIRCO, compare alternative compressor architectures, and improve the clarity and completeness of the method description.

All revisions are highlighted in blue in the revised manuscript.


W1 - 1: Baseline Reproduction

We thank the reviewer for raising concerns about baseline reproduction and DART’s reported improvements. We would like to clarify our experimental settings and the basis for our comparisons.

Baseline reproduction and evaluation. We reproduced ENCODER, OFFSET, and COMBINER using their official GitHub implementations, modifying the training scripts and corresponding model files as needed while retaining the official data splits and evaluation protocols. We followed the official evaluation protocols for FashionIQ, CIRR, and Shoes. For comparisons across frameworks, we used CLIP ViT-H/14 and a batch size of 32 for all three frameworks. As shown in the table below, each reproduced baseline and its DART counterpart share the same training configurations and evaluation protocols, enabling a fair and controlled assessment of DART's contribution.

FrameworkSettingOriginal PaperReproduction / DART
ENCODERCLIP BackboneViT-B/32ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate5e-55e-5
CLIP Learning Rate1e-61e-6
Batch Size12832
Epochs1010
PP / EE4 / 34 / 3
τ\tau / κ\kappa / γ\gamma / μ\mu0.1 / 0.8 / 0.5 / 0.50.1 / 0.8 / 0.5 / 0.5
OFFSETCLIP BackboneViT-H/14ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate1e-41e-4
CLIP Learning Rate1e-61e-6
Batch Size1632
Epochs1010
μ\mu (FashionIQ / Shoes / CIRR)0.5 / 0.5 / 0.80.5 / 0.5 / 0.8
PP / τ\tau4 / 0.14 / 0.1
COMBINERCLIP BackboneViT-H/14ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate1e-41e-4
CLIP Learning Rate1e-61e-6
Batch Size3232
Epochs1010
XX / YY / UU / HH4 / 4 / 8 / 19004 / 4 / 8 / 1900
ρ\rho / κ\kappa / μ\mu1.6 / 0.5 / 0.51.6 / 0.5 / 0.5
The framework-specific hyperparameter symbols follow the definitions in their respective original papers.

We also evaluated the officially released ENCODER checkpoint through our CIRR evaluation pipeline and reproduced its reported performance, providing additional evidence that our evaluation follows the official protocol. We acknowledge that our retrained baselines perform below the published results, particularly on CIRR. Changes in backbone or batch size may contribute to the gaps for some baselines, although we have not isolated their effects.

Upon acceptance, we will release the complete code, checkpoints, and environment configurations to support reproducibility and independent verification.

AuthorsOpenReview ↗

Response to Reviewer US3C (W1 - 2): Performance Improvements

DART's performance improvements. We understand the reviewer's concern about the magnitude of the reported improvements. To assess DART’s incremental contribution, we compare each reproduced baseline with its DART counterpart under identical training and evaluation conditions. This controlled comparison allows us to measure the performance gains from adding DART, whereas comparisons with published scores do not isolate this contribution.

We report the performance changes across three independent runs in the table below. DART improves retrieval performance across all three frameworks on FashionIQ and CIRR, with gains of 1.26–1.50 points in FashionIQ average R@10 and 0.20–2.02 points in CIRR average recall. On Shoes, DART improves ENCODER and OFFSET, while COMBINER shows a slight decline.

FrameworkFashionIQ ΔAvg. R@10FashionIQ ΔAvg. R@50CIRR ΔAvg. RecallShoes ΔAvg. Recall
COMBINER+1.35+0.71+0.20-0.36
ENCODER+1.26+0.51+0.65+1.99
OFFSET+1.50+1.04+2.02+1.82
Δ denotes the performance change after adding DART to each reproduced baseline, calculated from the mean results over three independent runs. All values are in percentage points.
AuthorsOpenReview ↗

Response to Reviewer US3C (W2–W3): Explanation of Eq. (8), Dataset Limitations, and Additional Evaluation

W2: Explanation of Eq. (8)

Thank you for pointing out the need to clarify the meaning of the variables in Eq. (8).

Variable definitions and roles. In our notation, ρ=K/N\rho=K/N is the compression prior, which provides a reference for token-wise allocation, and pip_i is the token-wise allocation coefficient. The paper does not define ρi\rho_i or refer to pip_i as a compression factor. TokenLearner was primarily designed for unimodal token compression. To further adapt it to CIR, DART introduces pip_i to adjust the original TokenLearner weights based on the modification instruction. We agree that this mechanism adjusts attention weights. The distinction is that pip_i applies token-specific reweighting, whereas a shared temperature applies the same scaling factor to all attention logits.

Motivation for the compression prior and KL regularization. Since TokenLearner compresses NN input tokens into KK compact tokens, we use ρ=K/N\rho=K/N as a uniform allocation prior. The KL term penalizes deviations of pip_i from this prior, discouraging excessive reweighting that could suppress other useful reference-image information.


W3: Dataset Limitations and Additional Evaluation

We thank the reviewer for highlighting the limitations of existing CIR benchmarks and suggesting evaluation on alternative datasets.

Limitations and Value of Existing Benchmarks. We acknowledge that some queries may admit text-only shortcuts and that incomplete positive annotations can introduce false negatives. In our FashionIQ qualitative results, for example, some unlabeled candidates retrieved by DART also appear to satisfy the modification instructions.

However, FashionIQ also requires distinguishing colors, styles, and other attributes within the same clothing category, for which recognizing the main object category alone is insufficient. We use FashionIQ and CIRR for controlled comparisons with existing supervised CIR methods under established evaluation protocols, while recognizing that these benchmarks alone cannot fully assess our approach.

Evaluation on i-CIR and CIRCO. CIRCO addresses some limitations of single-positive evaluation through multiple positive annotations. i-CIR emphasizes instance-level retrieval, requiring models to preserve object identity and jointly match the reference image and modification text. Following the reviewer’s suggestion, we evaluate CIRR-trained ENCODER, COMBINER, and their DART variants on both datasets (Appendix A.1, Tables 7 and 8).

As shown in tables below, DART improves all reported metrics for both frameworks on i-CIR and CIRCO. The gains are larger for ENCODER, while those for COMBINER on the CIRCO test set are relatively small. These results support DART’s effectiveness beyond FashionIQ and CIRR, although the gains vary across frameworks and datasets.

We also provide successful and failed retrieval cases on i-CIR (Appendix A.3, Figure 8). The successful cases illustrate better matching of object identity and modification requirements. The failed cases reveal a remaining limitation in fine-grained discrimination of visual attributes, such as illumination conditions.

FrameworkmAPmmAPminmAP
ENCODER37.8035.8212.62
ENCODER + DART39.9638.3613.76
COMBINER19.7920.135.47
COMBINER + DART20.6120.695.72

Cross-dataset retrieval results on i-CIR (%). All models are trained on CIRR and evaluated without additional fine-tuning.

ValidationTest
FrameworkmAP@5mAP@10mAP@25mAP@50mAP@5mAP@10mAP@25mAP@50
ENCODER22.2823.3025.2226.1321.9122.4024.4225.32
ENCODER + DART25.3426.1328.1929.1323.9624.5326.6727.58
COMBINER21.2622.0523.9524.8020.2920.8722.6323.55
COMBINER + DART22.3623.0324.8625.7020.4220.8922.7323.58

Cross-dataset retrieval results on CIRCO (%). All models are trained on CIRR and evaluated without additional fine-tuning.

Reviewer_1dAwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a common formulation of many supervised methods for CIR comprised of modality-specific encoders initialized from pretrained, and a method-specific local component that fuses local evidence with the global features. The paper is built on the result that the global representation is the dominant retrieval anchor, motivating the replacement of complicated, specialized local modules with a compact instruction-conditioned compression module. The resulting pipeline replacing parts of three global-local fusion baselines matches or improves retrieval on FIQ an CIRR.

Critical strengths

The paper tackles a nice question that challenges the increasingly heavy contrastively aligned architectures used for CIR. It provides nice insight into what parts of existing pipelines actually matter to inform practical designs for deployable models.

Results are stabilized over runs.

The proposed method seems lightweight and portable to a number of common methods for supervised CIR.

Critical weaknesses

The regularization design is not well motivated or sufficiently discussed. Since the optimum is a text-agnostic uniform allocation this seems like it would dampen instruction-conditioned learning. Last, since the paper doesn't use sampling from the DIB, this choice generally seems unmotivated.

It seems like the reproduced trainings across the three baselines consistently underperform published numbers. While degradation on FIQ is relatively minor, it's strange that results on CIRR are significantly worse than all papers' own metrics. Is there any non-standard evaluation used for CIRR? When comparing DART + {COMBINER, OFFSET, ENCODER} to the CIRR performance of published methods this work scores significantly lower. Also, since all baselines were reproduced it should be quite trivial to benchmark models trained on CIRR additionally on CIRCO, a standard benchmark in CIR literature.

The only ladder-style ablation given in Table 2 gives generally weak evidence pointing to all of the things this framework changes about baselines. Authors should consider supplying additional ablations similar in the Appendix on other backbone methods, or giving more discussion as to why its convincing to add so many additional modules for fractions of points of improvement.

The work motivates the learned token allocation as being instruction-specific but does not demonstrate this. The paper would benefit from showing that using a mismatched text for this step degrades performance, and empirical qualitative evidence of how allocation changes for the same image under different instructions. Also, the work's central motivation about token compression would benefit from a sweep over K.

Table 3 seems to measure computational cost in an unhelpful way. The local branch is expected to be a relatively minor cost relative to query encoding. Baselines also use captioning, segmentation, and heavy query encoders which raise questions regarding what computational model search is assumed to happen in.

The authors should consider discussion in context of previous work that (at a high level) are also learning modifications or reweightings over reference patches:

Questions for authors

See weaknesses

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1dAw (W1): DIB-inspired regularization

We sincerely appreciate the reviewer for the detailed and constructive feedback.

In response, we clarify the regularization design, baseline reproduction, and architectural components, and provide additional evaluations on CIRCO, analyses of modification-conditioned allocation, and comparisons with alternative architectures. We also revise the computational cost evaluation to better reflect full-query encoding.

All revisions are highlighted in blue in the revised manuscript.


W1: Motivation of DIB-inspired regularization

We thank the reviewer for highlighting the need to clarify the motivation of our regularizer.

The connection to DIB. DIB extracts relevant information from different input components under a limited information budget, with a separate constraint for each component. DART draws on this principle: TokenLearner compresses N input tokens into K compact tokens, and DART regularizes each input token’s allocation coefficient to guide how this limited local capacity is distributed. We describe DART as DIB-inspired because it adapts the component-wise constraint principle to deterministic token reweighting, whereas DIB uses stochastic sampling.

The motivation for regularization. The regularizer addresses a potential drawback of instruction-conditioned reweighting: placing too much emphasis on a few tokens may suppress other useful reference-image information. The KL term discourages excessive reweighting by penalizing large deviations from the allocation prior.

The text-agnostic uniform optimum of the KL term. We agree that the KL term alone is minimized when every allocation coefficient equals the prior, which recovers the original TokenLearner weights. During training, however, the retrieval loss encourages instruction-conditioned allocation, while the KL term limits excessive deviations from the prior. Our ablations and sensitivity analysis of β\beta show that moderate regularization improves retrieval performance.

AuthorsOpenReview ↗

Response to Reviewer 1dAw (W2 - 1): Baseline Reproduction

W2 - 1: Baseline Reproduction

We thank the reviewer for raising concerns about baseline reproduction.

Baseline reproduction and CIRR evaluation. We reproduced ENCODER, OFFSET, and COMBINER using their official GitHub implementations. We modified the training scripts and corresponding model files as needed, while retaining the original data loading and evaluation code. For FashionIQ, CIRR, and Shoes, we followed the official evaluation protocols. To enable comparisons across frameworks, we used CLIP ViT-H/14 and a batch size of 32 for all three frameworks. As shown in the table below, each reproduced baseline and its DART counterpart use identical training configurations and evaluation protocols, enabling a fair and controlled assessment of DART’s contribution.

FrameworkSettingOriginal PaperReproduction / DART
ENCODERCLIP BackboneViT-B/32ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate5e-55e-5
CLIP Learning Rate1e-61e-6
Batch Size12832
Epochs1010
PP / EE4 / 34 / 3
τ\tau / κ\kappa / γ\gamma / μ\mu0.1 / 0.8 / 0.5 / 0.50.1 / 0.8 / 0.5 / 0.5
OFFSETCLIP BackboneViT-H/14ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate1e-41e-4
CLIP Learning Rate1e-61e-6
Batch Size1632
Epochs1010
μ\mu (FashionIQ / Shoes / CIRR)0.5 / 0.5 / 0.80.5 / 0.5 / 0.8
PP / τ\tau4 / 0.14 / 0.1
COMBINERCLIP BackboneViT-H/14ViT-H/14
OptimizerAdamWAdamW
Initial Learning Rate1e-41e-4
CLIP Learning Rate1e-61e-6
Batch Size3232
Epochs1010
XX / YY / UU / HH4 / 4 / 8 / 19004 / 4 / 8 / 1900
ρ\rho / κ\kappa / μ\mu1.6 / 0.5 / 0.51.6 / 0.5 / 0.5
The framework-specific hyperparameter symbols follow the definitions in their respective original papers.

We also evaluated the officially released ENCODER checkpoint through our CIRR evaluation pipeline and reproduced its reported performance. This provides additional evidence that our CIRR evaluation follows the official protocol. We acknowledge that our retrained baselines achieve lower performance than the published results, particularly on CIRR. Changes in backbone or batch size may contribute to the gaps for some baselines, although their precise effects remain unclear.

Upon acceptance, we will release the complete code, checkpoints, and environment configurations to support reproduction and independent verification.

AuthorsOpenReview ↗

Response to Reviewer 1dAw (W2 - 2): Additional Evaluation

W2 - 2: Additional Evaluation

Additional Evaluation on CIRCO: Following the reviewer's suggestion, we evaluate the CIRR-trained models on CIRCO in Appendix A.1 (Table 8). Due to the additional segmentation preprocessing required by OFFSET, we report CIRCO results for ENCODER and COMBINER within the rebuttal period.

As shown in the table below, DART consistently improves all four mAP metrics for both frameworks.

ValidationTest
FrameworkmAP@5mAP@10mAP@25mAP@50mAP@5mAP@10mAP@25mAP@50
ENCODER22.2823.3025.2226.1321.9122.4024.4225.32
ENCODER + DART25.3426.1328.1929.1323.9624.5326.6727.58
COMBINER21.2622.0523.9524.8020.2920.8722.6323.55
COMBINER + DART22.3623.0324.8625.7020.4220.8922.7323.58

These results have been added to the Appendix of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback and their recognition of our work. For simplicity, we refer to Reviewers 1dAw, US3C, and GK2G as R1, R2, and R3, respectively.

Below, we summarize our contributions, clarifications, and experimental updates. All manuscript revisions are highlighted in blue.


Core Contributions

  • Empirical Findings. We show through controlled analyses that compact local modeling can retain most of the retrieval performance under a strong global anchor, motivating a shift toward how limited local capacity should be allocated.

  • Methodological Contribution. We propose DART, a lightweight local evidence allocation module that adapts reference-token weights within a compact KK-token representation according to modification semantics, together with DIB-inspired allocation regularization and balanced global--local integration.

  • Experimental Validation. We evaluate DART across three representative CIR frameworks on FashionIQ, CIRR, and Shoes, with cross-dataset evaluations on i-CIR and CIRCO, demonstrating comparable or improved retrieval performance with a lightweight local modeling design. Beyond retrieval performance, we conduct two sets of ablation studies to validate our design choices, sensitivity analyses of two key hyperparameters, and three quantitative and one qualitative analysis to investigate the allocation mechanism. We also provide qualitative retrieval comparisons on three datasets to illustrate the effectiveness and limitations of DART. We also evaluate computational costs to assess the efficiency of DART.


Clarifications and Manuscript Revisions

  1. Baseline Reproduction and Fair Comparisons (R1-W2, R2-W1). We clarify the reproduction settings and evaluation protocols, provide detailed hyperparameter comparisons, and verify the CIRR evaluation pipeline using the official ENCODER checkpoint. We also prepare the reproduced CIRR checkpoints for all three baselines to facilitate independent verification (Individual Responses).

  2. Regularization Design and Methodological Motivation (R1-W1/W5, R2-W2). We clarify the motivation and formulation of DIB-inspired regularization, explain why explicit sampling is not an inherent requirement of DIB, and discuss the rationale for lightweight local modeling (Individual Responses).

  3. Related Work and Architectural Design (R1-W3/W6). We extend the discussion of SAC and MELT, two modification-guided visual modeling approaches recommended by R1 (Section 2), and reorganize the component analysis to distinguish lightweight baseline construction from DART's allocation mechanism (Section 5.2, Table 2).

  4. Self-Containment and Architectural Details (R2-W5). We clarify the host framework background (Introduction, Section 3.2, Section 4.3), technical concepts (Introduction), and architecture of FθF_\theta (Section 4.2).


Updates to Experimental Results during Rebuttal

  1. Expanded Dataset Evaluation (R1-W2, R2-W3, R3-Q3).
  • Shoes (Appendix A.1, Table 6): We extend the evaluation to Shoes across all three frameworks.
  • i-CIR and CIRCO (Appendix A.1, Tables 7-8): We conduct cross-dataset evaluations of CIRR-trained ENCODER, COMBINER, and their DART variants.
  1. Modification-Conditioned Token Allocation Analysis (R1-W4).
  • Quantitative analysis (Section 5.3, Figure 3): We examine the impact of mismatched modification instructions on retrieval performance.
  • Qualitative analysis (Section 5.3, Figure 4): We visualize token-wise allocation for the same reference image under different modification instructions.
  1. Extended Hyperparameter Sensitivity Analysis (R1-W4, R3-Q1).
  • K-sensitivity (Appendix A.2, Figure 6(b)): We compare ENCODER, TL+BA, and DART across different values of KK.
  • β\beta-sensitivity (Appendix A.2, Figure 6(a)): We extend the sensitivity analysis of β\beta to all three host frameworks on FashionIQ.
  1. Additional Component and Architectural Ablations (R1-W3/W6, R2-W4).
  • Component ablations (Section 5.2, Table 2): We extend the component ablations to CIRR.

  • Alternative architectures (Appendix A.2, Table 9): We compare alternative compressor and modification-conditioning designs, including learnable-query cross-attention and SAC-/MELT-inspired alternatives.

  1. Revised Computational Cost Evaluation (R1-W5). We report full-query GFLOPs and latency alongside local-branch-specific parameter counts (Section 5.3, Table 3).

  2. Qualitative Retrieval Comparisons (R2-W3, R3-Q2). We provide successful and failed retrieval cases on FashionIQ, CIRR, and i-CIR to illustrate the effectiveness and limitations of DART (Section 5.4, Figure 5; Appendix A.3, Figures 7-8).


We look forward to further discussion and sincerely thank the reviewers and Area Chairs for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.