- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies mixed-modality retrieval, where text, image, and fused text-image documents are ranked together in the same corpus. The authors observe that existing retrievers are sensitive to the modality composition of the corpus, and in particular, irrelevant text documents can often receive higher similarity scores than relevant images. They refer to this phenomenon as “Chaos in the Text” and attribute it to modality preference. To address it, the paper proposes Trident, which constructs text, image, and fused views of each document as multiple positives and trains them with a shared Multi-Positive View InfoNCE objective.
Critical strengths
1. Clear Presentation.
The paper is generally well written and easy to follow. The problem formulation, phenomenon analysis, and proposed solution are organized in a clear progression.
2. Comprehensive Experimental Study.
The paper evaluates a broad range of CLIP-based and VLM-based retrievers, covers both visual-document and natural-image retrieval, and includes analyses on modality composition, distractor types, score distributions, and ablations.
3. Interesting Observation on Modality Preference.
The paper identifies a meaningful failure mode in mixed-modality retrieval: text documents can systematically receive higher scores and interfere with relevant visual evidence. The analysis goes beyond reporting performance drops and directly examines modality-dependent score differences and ranking interference.
Critical weaknesses
1. Comparability of Cross-Modal and Intra-Modal Similarity Scores
The analysis directly compares text–text and text–image similarities within a unified ranking space. However, for CLIP-style models, the training objective mainly enforces image–text alignment and does not explicitly guarantee that text–text and text–image similarity scores are calibrated on the same scale. Therefore, the observed preference for text documents may partly reflect the original training formulation of these models rather than a newly identified retrieval bias.
2. Fairness of the Fused-Document Evaluation
For models without native joint image–text encoding, the paper constructs fused representations by equally averaging image and text embeddings. This fusion operator is not necessarily optimized by the original models and may alter or suppress discriminative information. Since Trident is explicitly trained with such fused views while the original baselines are not, the comparison may partly reflect adaptation to the chosen fusion strategy rather than modality-preference mitigation alone.
3. Limited Methodological Novelty
A substantial part of the performance gain comes from explicitly introducing text, image, and fused positive views during training, whereas the original baselines are not trained for such mixed-modality settings. Multi-Positive View InfoNCE is a reasonable extension of contrastive learning, but the overall framework mainly aligns the training data and objective with the target evaluation scenario. The ablation also shows that modality-balanced data construction alone already contributes a large portion of the improvement.
4. Insufficient Analysis of the Source of Modality Preference
The paper clearly demonstrates modality-dependent score differences, but provides limited analysis of why this bias emerges. It remains unclear how much of the effect is caused by pretraining data imbalance, model architecture, training objectives, or modality-specific encoders. The evaluation could also be strengthened by including more methods specifically designed for joint image–text or composed-document retrieval.
Questions for authors
- How do the authors justify directly comparing text–text and text–image similarity scores, especially for CLIP-style models that are not explicitly trained to calibrate these two types of similarities on the same scale?
- For models without native joint image–text encoding, why is equal-weight embedding averaging used to construct fused documents? Since the baselines are not originally optimized for this fusion operator while Trident is trained with fused views, how can the authors ensure that the observed gains are not mainly due to adaptation to this specific fusion strategy?
5 comments
Response to Weakness 2 & Question 2 (Fairness of the fused-document evaluation)
We thank the reviewer for the comment. We respectfully disagree that the comparison is unfair. The limitation comes from CLIP-style models themselves, not from our evaluation design.
1. CLIP-style models have no native joint image–text encoding, which is a limitation of the paradigm itself. A CLIP-style dual-tower model can only encode text and images separately. For a fused document to enter the same candidate pool, the two vectors must be combined at the output. The 0.5/0.5 average is the simplest option, introducing no extra parameters and relying on no extra assumptions. We apply the same operator to all CLIP-style baselines (including the GR-CLIP variants) and to Trident-JinaCLIP, so the evaluation conditions are identical. Mixed-modality corpora naturally contain interleaved image–text documents, and being unable to handle them natively is itself a shortcoming of CLIP-style models in this setting. The reviewer's point that the fusion operator was not optimized for the original model is precisely the problem with these models, not an extra factor introduced by the evaluation. Likewise, Trident teaching the model to handle the fused view during training is part of the method, not an attempt to game the evaluation operator.
2. Our core findings do not depend on fused documents.
- Chaos in the Text (Fig. 3) compares only irrelevant-text and irrelevant-image distractors and involves no fused documents.
- The "Doc: Text" vs. "Doc: Image" comparison in Fig. 4b likewise requires no fusion operator.
- The V-shape in Fig. 2 comes from the coexistence of text and image documents. Increasing the proportion of fused documents does not produce a V-shape and in fact alleviates the degradation.
3. For VLMs, no external fusion operator is used. Qwen3-VL-Embedding and Trident-Qwen3VL-2B both encode fused documents natively and jointly (Table 2), so the 0.5/0.5 average does not apply. In this setting, Mix Avg is 73.90 vs. 40.12, and the gain cannot be attributed to adaptation to a particular fusion operator.
4. Additional experiment: evaluation on a mixed corpus without fused documents. We take a mixed corpus with a fusion ratio of 0 and a text ratio of 0.5 (containing only text and image documents) and directly compare each baseline with Trident, without modifying any model. The 0.5/0.5 average plays no role in this evaluation.
| Model | Mix Avg (no fused documents) |
|---|---|
| Qwen3-VL-E-2B | 43.54 |
| Trident-Qwen3VL-2B | 71.93 (+28.39) |
| Jina-CLIP-v2 | 36.15 |
| Trident-JinaCLIP | 52.90 (+16.75) |
On a corpus with no fused documents at all, Trident still outperforms its baseline by a large margin on both backbones, and the gains are of the same order as when fused documents are included. This indicates that the improvement comes mainly from alleviating the modality preference between text and images, rather than from adapting to a particular fusion operator.
We will add this experiment to the appendix and note that the lack of native joint encoding is one reason for the limited performance of CLIP-style models on fused documents.
Response to Weakness 3 (Methodological novelty)
We agree that multi-positive contrastive learning is not a new paradigm in itself, and that the three-view data contributes part of the gain. However, we disagree with the characterization that the overall framework merely aligns the data and objective to the evaluation setting, for three reasons.
1. Our core contribution is discovering and characterizing "modality preference." Existing work (e.g., GR-CLIP) focuses on the modality gap, i.e., the geometric offset between image and text embeddings, which cancels out in the ranking when all candidates share one modality. We identify a different problem: when text, image, and fused documents compete in the same candidate pool, the model assigns systematically different scores to semantically equivalent documents depending on their modality, so that irrelevant texts outrank relevant images in 76.0%–94.6% of cases (Fig. 4b). This phenomenon is present in more than nine retrievers and persists after GR-CLIP correction (Appendices A–C). Trident is the solution we designed for this finding.
2. The contributions of data and objective can be separated, and both matter. The controls below use the same initialization, the same source of training data, and the same hyperparameters (Avg is the mean over the six benchmarks in Table 1):
| Setting | Origin Avg | Mix Avg |
|---|---|---|
| Qwen3-VL-E-2B (off-the-shelf) | 77.59 | 40.12 |
| Control 1: single view + InfoNCE | 77.36 | 66.36 |
| Control 2: three views + per-modality independent normalization | 79.76 | 70.63 |
| Trident: three views + shared normalization | 80.51 | 73.90 |
- Fine-tuning alone raises Mix from 40.12 to 66.36 while Origin is essentially unchanged, so this step mainly adapts the model to the mixed corpus and does not mean modality preference has been removed. We do not downplay this.
- On top of that, three-view data adds +4.27 Mix (control 1 → control 2), and shared normalization adds a further +3.27 on identical data (control 2 → Trident), with Origin improving as well. Both contribute.
- The baselines also do not lack training data: Qwen3-VL-Embedding-8B reaches a Mix Avg of only 47.17, more than 26 points below the 2B Trident (73.90). Existing training recipes therefore do not automatically yield cross-modally comparable scores.
3. Shared normalization has a clear theoretical basis. Independent normalization is insensitive to a constant offset applied to all candidates within one modality (Eq. 3), so it cannot address the score baseline gap between modalities. Shared normalization places the three positives and the negatives in a single denominator, which constrains such offsets (Appendix E.5).
In the revision, we will include this data-versus-objective breakdown in the main text.
Response to Weakness 1 & Question 1 (Comparability of cross-modal and same-modal similarity scores)
We thank the reviewer for the comment. We respectfully disagree that this comparison is unfair.
1. Mixed-modality retrieval is a basic real-world need, and cross-modal score comparability is a basic requirement it places on the model. Real knowledge bases rarely contain a single modality. Web pages and encyclopedia entries are mostly text, key figures in annual reports, research reports, and slides often exist only in charts or page screenshots, and enterprise document stores mix plain text, plain images, and interleaved files. In multimodal RAG and deep-research applications, the user enters a single text query and the system must find evidence in such a mixed corpus, where the evidence may be in any modality. The system cannot know in advance which modality holds the answer, and it should not maintain a separate index per modality, take Top-K from each, and merge them by hand. The usual practice is therefore to put all documents in one candidate pool and rank them uniformly by sim(q, d) (Eq. 2).
Whether scores are comparable across modalities is thus not an assumption we added. It is a basic requirement this paradigm places on the retriever. CLIP-style objectives constrain only image–text alignment and do not calibrate the score scales of text–text and text–image similarity, which shows these models have a problem under this paradigm. That problem is exactly the phenomenon we characterize. The cause may well lie in the training formulation, but this does not change the fact that, in a mixed corpus, such models push relevant images out of the Top-K in favor of irrelevant texts (Fig. 1, Fig. 3, Fig. 4b).
2. If this were only a result of CLIP's training formulation, it should not appear in VLM-based models. The VLM-based retrievers we evaluate (Qwen3-VL-Embedding-2B/8B, UniME-Phi3.5, Jina-embeddings-v5-omni-small) encode text and images jointly with a unified Transformer backbone rather than CLIP-style dual-tower contrastive training. Yet they show a preference in the same direction (Fig. 4, Fig. 12):
In addition, the V-shaped curves in Figs. 2, 8, and 9 and the finding in Figs. 3 and 10 that text distractors are far more harmful than image distractors hold for both CLIP-style and VLM-based models. The same directional preference appearing across architectures, training objectives, and model scales indicates a widespread phenomenon rather than a flaw specific to one model family.
3. Simply calibrating the score scale does not remove the problem. In Appendix C (Fig. 11), for CLIP-style models calibrated with GR-CLIP post-processing, relevant images are still outranked by irrelevant texts in 70.6%–82.6% of cases. If the issue were only a misaligned scale between text–text and text–image similarity, calibration should largely eliminate it.
4. The specific form of the preference depends on the architecture. Fig. 7 shows that Jina-CLIP-v2 prefers Image–Image similarity, whereas Qwen3-VL-Embedding-2B prefers Text–Text. The behavior differs across models and is hard to attribute to a single training formula.
5. The preference can be removed through training. After training with three-view equal positives and shared normalization, the positive similarities of the three views in Trident-Qwen3VL-2B converge (0.674 / 0.672 / 0.670), and the interference rates drop to 10.4% / 10.6% / 11.8% (Fig. 12). This shows it is not an unavoidable scale issue but one that retrieval training can correct.