acceptodds
Under review as a conference paper at ICLR 2027

Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Abstract

Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term ***Chaos in the Text***. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce **Trident**, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Retrievers, Mixed-Modality Retrieval, Modality Preference, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dAPROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies mixed-modality retrieval, where text, image, and fused text-image documents are ranked together in the same corpus. The authors observe that existing retrievers are sensitive to the modality composition of the corpus, and in particular, irrelevant text documents can often receive higher similarity scores than relevant images. They refer to this phenomenon as “Chaos in the Text” and attribute it to modality preference. To address it, the paper proposes Trident, which constructs text, image, and fused views of each document as multiple positives and trains them with a shared Multi-Positive View InfoNCE objective.

Critical strengths

1. Clear Presentation.

The paper is generally well written and easy to follow. The problem formulation, phenomenon analysis, and proposed solution are organized in a clear progression.

2. Comprehensive Experimental Study.

The paper evaluates a broad range of CLIP-based and VLM-based retrievers, covers both visual-document and natural-image retrieval, and includes analyses on modality composition, distractor types, score distributions, and ablations.

3. Interesting Observation on Modality Preference.

The paper identifies a meaningful failure mode in mixed-modality retrieval: text documents can systematically receive higher scores and interfere with relevant visual evidence. The analysis goes beyond reporting performance drops and directly examines modality-dependent score differences and ranking interference.

Critical weaknesses

1. Comparability of Cross-Modal and Intra-Modal Similarity Scores

The analysis directly compares text–text and text–image similarities within a unified ranking space. However, for CLIP-style models, the training objective mainly enforces image–text alignment and does not explicitly guarantee that text–text and text–image similarity scores are calibrated on the same scale. Therefore, the observed preference for text documents may partly reflect the original training formulation of these models rather than a newly identified retrieval bias.

2. Fairness of the Fused-Document Evaluation

For models without native joint image–text encoding, the paper constructs fused representations by equally averaging image and text embeddings. This fusion operator is not necessarily optimized by the original models and may alter or suppress discriminative information. Since Trident is explicitly trained with such fused views while the original baselines are not, the comparison may partly reflect adaptation to the chosen fusion strategy rather than modality-preference mitigation alone.

3. Limited Methodological Novelty

A substantial part of the performance gain comes from explicitly introducing text, image, and fused positive views during training, whereas the original baselines are not trained for such mixed-modality settings. Multi-Positive View InfoNCE is a reasonable extension of contrastive learning, but the overall framework mainly aligns the training data and objective with the target evaluation scenario. The ablation also shows that modality-balanced data construction alone already contributes a large portion of the improvement.

4. Insufficient Analysis of the Source of Modality Preference

The paper clearly demonstrates modality-dependent score differences, but provides limited analysis of why this bias emerges. It remains unclear how much of the effect is caused by pretraining data imbalance, model architecture, training objectives, or modality-specific encoders. The evaluation could also be strengthened by including more methods specifically designed for joint image–text or composed-document retrieval.

Questions for authors
  1. How do the authors justify directly comparing text–text and text–image similarity scores, especially for CLIP-style models that are not explicitly trained to calibrate these two types of similarities on the same scale?
  2. For models without native joint image–text encoding, why is equal-weight embedding averaging used to construct fused documents? Since the baselines are not originally optimized for this fusion operator while Trident is trained with fused views, how can the authors ensure that the observed gains are not mainly due to adaptation to this specific fusion strategy?
5 comments

Response to Weakness 1 & Question 1 (Comparability of cross-modal and same-modal similarity scores)

We thank the reviewer for the comment. We respectfully disagree that this comparison is unfair.

1. Mixed-modality retrieval is a basic real-world need, and cross-modal score comparability is a basic requirement it places on the model. Real knowledge bases rarely contain a single modality. Web pages and encyclopedia entries are mostly text, key figures in annual reports, research reports, and slides often exist only in charts or page screenshots, and enterprise document stores mix plain text, plain images, and interleaved files. In multimodal RAG and deep-research applications, the user enters a single text query and the system must find evidence in such a mixed corpus, where the evidence may be in any modality. The system cannot know in advance which modality holds the answer, and it should not maintain a separate index per modality, take Top-K from each, and merge them by hand. The usual practice is therefore to put all documents in one candidate pool and rank them uniformly by sim(q, d) (Eq. 2).

Whether scores are comparable across modalities is thus not an assumption we added. It is a basic requirement this paradigm places on the retriever. CLIP-style objectives constrain only image–text alignment and do not calibrate the score scales of text–text and text–image similarity, which shows these models have a problem under this paradigm. That problem is exactly the phenomenon we characterize. The cause may well lie in the training formulation, but this does not change the fact that, in a mixed corpus, such models push relevant images out of the Top-K in favor of irrelevant texts (Fig. 1, Fig. 3, Fig. 4b).

2. If this were only a result of CLIP's training formulation, it should not appear in VLM-based models. The VLM-based retrievers we evaluate (Qwen3-VL-Embedding-2B/8B, UniME-Phi3.5, Jina-embeddings-v5-omni-small) encode text and images jointly with a unified Transformer backbone rather than CLIP-style dual-tower contrastive training. Yet they show a preference in the same direction (Fig. 4, Fig. 12):

ModelText-positive similarityImage-positive similarityText hard-negative interference rate (text positive / image positive)
Jina-CLIP-v2 (CLIP)0.5310.30825.8% / 94.6%
Qwen3-VL-E-2B0.6640.48417.8% / 87.8%
Qwen3-VL-E-8B0.6310.44515.8% / 76.0%
UniME-Phi3.5-V-4.2B0.5930.51918.8% / 52.6%
Jina-V5-small0.6030.46112.8% / 53.2%

In addition, the V-shaped curves in Figs. 2, 8, and 9 and the finding in Figs. 3 and 10 that text distractors are far more harmful than image distractors hold for both CLIP-style and VLM-based models. The same directional preference appearing across architectures, training objectives, and model scales indicates a widespread phenomenon rather than a flaw specific to one model family.

3. Simply calibrating the score scale does not remove the problem. In Appendix C (Fig. 11), for CLIP-style models calibrated with GR-CLIP post-processing, relevant images are still outranked by irrelevant texts in 70.6%–82.6% of cases. If the issue were only a misaligned scale between text–text and text–image similarity, calibration should largely eliminate it.

4. The specific form of the preference depends on the architecture. Fig. 7 shows that Jina-CLIP-v2 prefers Image–Image similarity, whereas Qwen3-VL-Embedding-2B prefers Text–Text. The behavior differs across models and is hard to attribute to a single training formula.

5. The preference can be removed through training. After training with three-view equal positives and shared normalization, the positive similarities of the three views in Trident-Qwen3VL-2B converge (0.674 / 0.672 / 0.670), and the interference rates drop to 10.4% / 10.6% / 11.8% (Fig. 12). This shows it is not an unavoidable scale issue but one that retrieval training can correct.

Response to Weakness 2 & Question 2 (Fairness of the fused-document evaluation)

We thank the reviewer for the comment. We respectfully disagree that the comparison is unfair. The limitation comes from CLIP-style models themselves, not from our evaluation design.

1. CLIP-style models have no native joint image–text encoding, which is a limitation of the paradigm itself. A CLIP-style dual-tower model can only encode text and images separately. For a fused document to enter the same candidate pool, the two vectors must be combined at the output. The 0.5/0.5 average is the simplest option, introducing no extra parameters and relying on no extra assumptions. We apply the same operator to all CLIP-style baselines (including the GR-CLIP variants) and to Trident-JinaCLIP, so the evaluation conditions are identical. Mixed-modality corpora naturally contain interleaved image–text documents, and being unable to handle them natively is itself a shortcoming of CLIP-style models in this setting. The reviewer's point that the fusion operator was not optimized for the original model is precisely the problem with these models, not an extra factor introduced by the evaluation. Likewise, Trident teaching the model to handle the fused view during training is part of the method, not an attempt to game the evaluation operator.

2. Our core findings do not depend on fused documents.

  • Chaos in the Text (Fig. 3) compares only irrelevant-text and irrelevant-image distractors and involves no fused documents.
  • The "Doc: Text" vs. "Doc: Image" comparison in Fig. 4b likewise requires no fusion operator.
  • The V-shape in Fig. 2 comes from the coexistence of text and image documents. Increasing the proportion of fused documents does not produce a V-shape and in fact alleviates the degradation.

3. For VLMs, no external fusion operator is used. Qwen3-VL-Embedding and Trident-Qwen3VL-2B both encode fused documents natively and jointly (Table 2), so the 0.5/0.5 average does not apply. In this setting, Mix Avg is 73.90 vs. 40.12, and the gain cannot be attributed to adaptation to a particular fusion operator.

4. Additional experiment: evaluation on a mixed corpus without fused documents. We take a mixed corpus with a fusion ratio of 0 and a text ratio of 0.5 (containing only text and image documents) and directly compare each baseline with Trident, without modifying any model. The 0.5/0.5 average plays no role in this evaluation.

ModelMix Avg (no fused documents)
Qwen3-VL-E-2B43.54
Trident-Qwen3VL-2B71.93 (+28.39)
Jina-CLIP-v236.15
Trident-JinaCLIP52.90 (+16.75)

On a corpus with no fused documents at all, Trident still outperforms its baseline by a large margin on both backbones, and the gains are of the same order as when fused documents are included. This indicates that the improvement comes mainly from alleviating the modality preference between text and images, rather than from adapting to a particular fusion operator.

We will add this experiment to the appendix and note that the lack of native joint encoding is one reason for the limited performance of CLIP-style models on fused documents.

Response to Weakness 3 (Methodological novelty)

We agree that multi-positive contrastive learning is not a new paradigm in itself, and that the three-view data contributes part of the gain. However, we disagree with the characterization that the overall framework merely aligns the data and objective to the evaluation setting, for three reasons.

1. Our core contribution is discovering and characterizing "modality preference." Existing work (e.g., GR-CLIP) focuses on the modality gap, i.e., the geometric offset between image and text embeddings, which cancels out in the ranking when all candidates share one modality. We identify a different problem: when text, image, and fused documents compete in the same candidate pool, the model assigns systematically different scores to semantically equivalent documents depending on their modality, so that irrelevant texts outrank relevant images in 76.0%–94.6% of cases (Fig. 4b). This phenomenon is present in more than nine retrievers and persists after GR-CLIP correction (Appendices A–C). Trident is the solution we designed for this finding.

2. The contributions of data and objective can be separated, and both matter. The controls below use the same initialization, the same source of training data, and the same hyperparameters (Avg is the mean over the six benchmarks in Table 1):

SettingOrigin AvgMix Avg
Qwen3-VL-E-2B (off-the-shelf)77.5940.12
Control 1: single view + InfoNCE77.3666.36
Control 2: three views + per-modality independent normalization79.7670.63
Trident: three views + shared normalization80.5173.90
  • Fine-tuning alone raises Mix from 40.12 to 66.36 while Origin is essentially unchanged, so this step mainly adapts the model to the mixed corpus and does not mean modality preference has been removed. We do not downplay this.
  • On top of that, three-view data adds +4.27 Mix (control 1 → control 2), and shared normalization adds a further +3.27 on identical data (control 2 → Trident), with Origin improving as well. Both contribute.
  • The baselines also do not lack training data: Qwen3-VL-Embedding-8B reaches a Mix Avg of only 47.17, more than 26 points below the 2B Trident (73.90). Existing training recipes therefore do not automatically yield cross-modally comparable scores.

3. Shared normalization has a clear theoretical basis. Independent normalization is insensitive to a constant offset applied to all candidates within one modality (Eq. 3), so it cannot address the score baseline gap between modalities. Shared normalization places the three positives and the negatives in a single denominator, which constrains such offsets (Appendix E.5).

In the revision, we will include this data-versus-objective breakdown in the main text.

Reviewer_sMjnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mixed-modality retrieval plays an important role in multimodal RAG. Dense retrievers have made notable progress on text and image corpora, but it remains unclear whether these capabilities extend reliably to mixed corpora containing text, images, and fused text–image documents. This paper therefore systematically compares retrievers of different architectures and finds that their performance is highly sensitive to the modality composition, with especially severe drops in the presence of irrelevant text. The authors attribute this to a modality preference, whereby models assign higher scores to text. They propose Trident, which constructs the text, image, and fused text–image representations of each document as three equal positives for the same query and optimizes them jointly with a multi-positive InfoNCE objective under a shared normalization. While distinguishing relevant from irrelevant documents, this objective prevents the positive probability from concentrating on a single modality, thereby mitigating modality preference. On six visual-document retrieval benchmarks, Trident raises the mixed-corpus NDCG@10 of Qwen3-VL-2B from 40.12 to 73.90, surpassing the 8B model, and also improves average single-modality performance.

Critical strengths
  1. The paper reveals a blind spot in existing evaluation. It shows that models performing well on single-modality benchmarks can fail markedly on mixed corpora; for example, the average score of Qwen3-VL-Embedding-8B drops from 83.47 to 47.17. This matters for my assessment because the finding is useful independently of the proposed method, and it is the main value I see in the paper.
Critical weaknesses
  1. Missing simple baselines and an unequal comparison. Simple baselines are missing, such as retrieving per modality and fusing with RRF, or normalizing scores per modality. In addition, Trident is fine-tuned on OpenDocVQA, while most baselines are off-the-shelf models, so the comparison is not entirely fair. This matters because, without these controls, it is unclear how much of the gain comes from the proposed objective rather than from fine-tuning or from a simple score calibration.
  2. Possible confound from same-source synthetic text. The text views in both the training data and the mixed test corpora are generated by Qwen3-VL-235B with the same pipeline, so the model may simply have learned stylistic features of the generator. On MixBench, which was built by a third party, its lead shrinks markedly. This matters because it leaves open whether Trident corrects a genuine modality preference, which is the paper's central claim.
Questions for authors

1.The text/fused views in both the training data and the mixed test corpora are generated by Qwen3-VL-235B with the same pipeline. How do you rule out that the model has learned the generator's style rather than a genuine modality preference? Does Chaos in the Text still occur on human-written or independently sourced text? 2.Trident is fine-tuned on OpenDocVQA, while most baselines are not trained on the same three-view data. Can you provide strict controls, such as fine-tuning only on the original single-modality data, or using the same data without the multi-positive InfoNCE objective?

2 comments

Response to Weakness 1 & Question 1 (Missing simple baselines and an unequal comparison)

We agree that the original submission did not separate the sources of the gain. It could come from the training objective itself, from fine-tuning on OpenDocVQA, or simply from score calibration. To address this, we added two sets of controls. Both use Qwen3-VL-2B as the backbone, with the same evaluation setup as Table 1.

1. Simple post-processing baselines

Table 1 already includes a post-hoc calibration method on the CLIP side (GR-CLIP). Here we add three post-processing methods on the VLM side: RRF fusion, per-modality Z-score, and per-modality Min-Max, all applied to the off-the-shelf Qwen3-VL-Embedding-2B.

Post-processingNoneRRFZ-scoreMin-Max
Mix Avg of Qwen3-VL-E-2B40.1249.9152.5348.27

Post-processing does help. The best variant, Z-score, raises Mix Avg from 40.12 to 52.53, which supports our argument that modality preference makes cross-modal scores incomparable. However, it still trails Trident (73.90) by more than 21 points. These methods also require knowing each document's modality and estimating per-modality statistics, which adds cost in real deployments.

2. Fine-tuning controls on identical data

To separate the benefit of fine-tuning from the benefit of the training objective, we trained controls with the same training data (OpenDocVQA), the same initialization (Qwen3-VL-2B-Instruct), and the same hyperparameters, changing only the positive-sample construction or the loss. The last column reports Mix Avg after applying the best post-processing method to each model.

ModelPositives / lossOrigin AvgMix AvgMix Avg (+ best post-processing)
Qwen3-VL-E-2BOff-the-shelf77.5940.1252.53
Control 1Single random view, InfoNCE77.3666.3668.34
Control 2Three views, per-modality independent normalization79.7670.6370.94
TridentThree views, shared normalization80.5173.9072.18 (slightly lower with post-processing)

One clarification: the "w/o Multi-Positive View InfoNCE" ablation in Table 1 is exactly Control 1. Each sample uses one randomly chosen view, modality proportions are balanced, and the loss is standard InfoNCE, so the numbers match. That ablation removed the three-view data and the shared normalization at the same time, so it could not isolate the effect of the objective. Control 2 fills this gap: it keeps the three-view data and replaces only the shared normalization with per-modality independent normalization. In the revision, we will state this setup clearly and add it to the ablation.

Our reading of the table is as follows:

  • Fine-tuning itself accounts for part of the gain. It raises Mix from 40.12 to 66.36 while leaving Origin essentially unchanged (77.59 → 77.36). So it mainly adapts the model to the mixed corpus, and does not show that modality preference has been removed.
  • Three-view data helps. With independent normalization in both cases, moving from a single view (Control 1) to three views (Control 2) raises Mix from 66.36 to 70.63 and Origin from 77.36 to 79.76.
  • Shared normalization adds a further gain. On exactly the same three-view data, switching from independent to shared normalization raises Mix from 70.63 to 73.90 (+3.27) and Origin from 79.76 to 80.51 (+0.75). Independent normalization is insensitive to a constant offset applied to all candidates within one modality (Eq. (3) and Appendix E.5), so it cannot address the score baseline gap between modalities. Shared normalization can. Relative to the original ablation (Control 1), the objective and data together give a +7.5 Mix improvement. With the two now separated, about 4.3 points come from the three-view data and about 3.3 from the objective itself.

3. Is post-processing still useful?

Models fine-tuned with independent normalization (Controls 1 and 2) still gain about +1 to +2 from post-processing, which indicates residual modality preference. Trident gains nothing from post-processing (Z-score: −0.45, RRF: −1.15, Min-Max: −2.20), which suggests that the per-modality score baselines have already been aligned during training. Even when every control is paired with its best post-processing, the highest result is 70.94 (Control 2 + Z-score), still below Trident without any post-processing (73.90).

Response to Weakness 2 & Question 2 (Possible confounding from same-source synthetic text)

We answer in three parts: we first rebuild the test corpus with a different generator, then point to evidence in the paper that does not depend on our generation pipeline, and finally explain why the margin on MixBench is smaller.

1. Using different generators for training and testing

We keep the model fixed (Trident is still trained on text views generated by Qwen3-VL-235B) and replace only the text views in the test corpus with versions generated by Qwen3.8-27B using the same pipeline (detailed descriptions and summaries). The writing style seen at training time therefore differs from that at test time.

ModelTest-corpus text generated byOrigin AvgMix Avg
Qwen3-VL-E-2BQwen3-VL-235B77.5940.12
Qwen3-VL-E-2BQwen3.8-27B-41.07
Trident-Qwen3VL-2BQwen3-VL-235B80.5173.90
Trident-Qwen3VL-2BQwen3.8-27B-74.00

With the 27B text, both the off-the-shelf model (40.12 → 41.07) and Trident (73.90 → 74.00) barely change, and the gap between them stays at roughly 33 points. If Trident had learned the writing style of the 235B generator, we would expect a clear drop when the text comes from a different generator. We do not observe one.

2. Evidence that does not depend on our generation pipeline

First, modality preference does not originate from the generator. The distractor texts in Figure 3 and Appendix B are real passages sampled from a Wikipedia corpus, not text we generated, yet they still cripple the off-the-shelf model. After Trident training, the gap caused by text distractors and by image distractors narrows markedly (Appendix B, Figure 10).

Second, the mixed corpus in MixBench was built by a third party (Li et al., 2025a), and its text does not come from our pipeline. Off-the-shelf baselines also degrade sharply on it (SigLIP2 drops from 53.13 to 21.81, CLIP-ViT-L/14 from 69.32 to 37.98, and VisRAG-Ret from 55.67 to 33.91). After Trident training, both backbones achieve the highest Mix average:

BackboneBaseline MixTrident MixGainWithout three views and shared normalization
Jina-CLIP-v256.43 (GR-CLIP: 64.06)70.50+14.07 (+6.44 over GR-CLIP)67.73
Qwen3-VL-2B73.6776.78+3.1174.02

Trident-Qwen3VL-2B also outperforms the larger Qwen3-VL-Embedding-8B (74.43). The ablation setup is the same as in Table 1, removing the three-view data and shared normalization together. Mix drops by about 2.8 points on both backbones, showing that these two components jointly still matter on a third-party corpus.

3. Why the margin on MixBench is smaller

The VLM-side gain on MixBench (+3.11) is indeed smaller than on the document corpus. The main reason is that Qwen3-VL-Embedding-2B already performs well on natural images (Mix 73.67, even higher than its Origin of 66.60), which leaves limited headroom for Trident. Meanwhile, the CLIP-side gain does not shrink on the third-party corpus (+12.83 on the document corpus vs. +14.07 on MixBench), so the smaller margin does not indicate that Trident fails once the text is no longer ours.

Reviewer_MVKcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies modality preference in mixed-modality retrieval. When text, image, and fused image-text documents are placed in a shared corpus, retrieval performance may decline, and irrelevant text can sometimes rank above relevant images. To address this issue, the authors propose Trident, which jointly trains on three views of the same document using multi-positive contrastive learning and improves performance on both visual-document and natural-image retrieval benchmarks.

Critical strengths
  1. The paper addresses a practical limitation of multimodal retrieval: good performance within individual modalities does not ensure reliable ranking in a mixed corpus. Its analysis across models, modality proportions, and distractor settings provides useful evidence of this problem and supports the relevance of the research question.
  2. Trident is straightforward to implement and improves average mixed-modality retrieval performance for both CLIP-based and VLM-based models, with particularly notable gains on visual-document benchmarks. These results suggest that jointly training document views is a useful approach to the identified problem.
Critical weaknesses
  1. The contribution beyond existing multi-positive and three-view contrastive learning methods [1, 2] could be made clearer. In particular, the paper does not yet sufficiently establish what Trident's query supervision adds relative to closely related methods such as GCL, which makes its methodological contribution difficult to assess.
  2. Differences in information across document views and unclear comparison and ablation settings leave some uncertainty about the experimental results. These issues make it difficult to determine how much of the observed behavior reflects modality preference and which training choices account for the reported gains.
Questions for authors
  1. Equation (4) takes the same form as the multi-positive supervised contrastive loss [1], and GCL [2] also trains text, image, and fused image-text views with shared normalization. Could the authors clarify how Trident differs in its query supervision and training procedure? Relevant comparisons would help explain the benefits of these differences.
  2. Generated descriptions may alter information relevant to the query, and Appendix G also notes differences in the information provided by each view. It would be helpful to discuss how well query-relevant information is preserved when constructing these views and how any differences might affect the main conclusions.
  3. According to Table 2, Trident is initialized from Qwen3-VL-2B-Instruct, whereas the main baseline uses Qwen3-VL-Embedding-2B. The training setup used when Multi-Positive View InfoNCE is removed is also unclear. Could the authors provide further details on the initialization, training data, loss functions, and sampling settings for these comparisons and ablations?
  4. With three positive samples sharing the same softmax denominator, Equation (4) has a theoretical lower bound of "log(3) ≈ 1.099" , but the retrieval loss in Figure 15(a) falls below this value. This may reflect a constant offset, rescaling, or the reporting of a different loss component. Could the authors clarify what the curve represents and how it relates to Equation (4)? References [1] Prannay Khosla et al. Supervised Contrastive Learning. NeurIPS, 2020. [2] Jungsoo Lee et al. Generalized Contrastive Learning for Universal Multimodal Retrieval. NeurIPS, 2025.
3 comments

Response to Weakness 1 & Question 1 (Contribution relative to GCL)

We thank the reviewer for the comment. First, a clarification: multi-positive contrastive learning and using multiple modality representations of the same sample are not novel to Trident, and GCL also uses three representations of the same sample. In the revision, we will cite these works explicitly in the method section and in Related Work, and state clearly how our work relates to them.

Our core contribution is the first systematic analysis of "modality preference" in mixed-modality retrieval. Trident is our solution to the problem this analysis uncovers.

1. First analysis of this problem. Existing work (including GCL and GR-CLIP) focuses on the modality gap, i.e., the geometric separation between image and text embeddings, and evaluates on alignment tasks whose candidates are all of a single modality. In that setting, the overall offset caused by the gap cancels out in the ranking. We are the first to systematically study corpora where text, image, and fused documents coexist, and our analysis yields the following findings:

  • As image documents are progressively replaced by semantically equivalent text, retrieval performance follows a clear V-shaped curve (Figures 1 and 2).
  • The same number of irrelevant texts hurts more than irrelevant images, which we call "Chaos in the Text" (Figure 3).
  • The root cause is an internal modality preference: for semantically equivalent relevant documents, the text view systematically receives a higher similarity than the image view, so irrelevant texts often outrank relevant images (Figure 4).
  • The phenomenon is present in more than nine retrievers and persists even after GR-CLIP corrects the modality gap (Appendices A–C). It is therefore distinct from the modality gap and cannot be fixed by geometric correction alone.

This finding also answers why query supervision is necessary. The phenomenon we want to correct is defined by query relevance: without a query, there is no such thing as an irrelevant text outranking a relevant image, so query-free alignment training cannot address it.

2. Trident's design follows directly from this finding. Using real query–document supervision, we construct three semantically equivalent views for each document and treat them as equal positives, with all three positives and the negatives sharing a single denominator. The objective decomposes into −log ρ + D_KL(u‖r) + log M (Eq. 6), and the gradient for each positive is (p_m − 1/M)/τ. A view whose probability is too high is pushed down, and a view whose probability is too low (e.g., a relevant image) is pushed up. High-scoring irrelevant texts, acting as negatives, are pushed down directly (Appendix E.3).

3. Controlled experiments. To separate the effect of query supervision from that of how the positives are coupled, we trained the following controls on the same three-view data built from OpenDocVQA, using Qwen3-VL-2B and the same hyperparameters as Trident. Avg is the mean over the six benchmarks in Table 1.

MethodTraining signalOrigin AvgMix Avg
Qwen3-VL-E-2B–77.5940.12
GCL-style, no queryThe three views of the same sample serve as anchor and positives for each other55.5344.00
GCL-style, with queryQuery + three views; each positive is a separate term, with the other positives masked77.4867.82
Three views + per-modality independent normalizationQuery + three views79.7670.63
TridentQuery + three views; the three positives share one denominator80.5173.90
  • Query supervision is necessary. Training without queries raises Mix only from 40.12 to 44.00 while Origin drops substantially. Aligning the three representations alone does not carry relevance into a unified ranking.
  • Query supervision plus three views accounts for most of the gain. Mix reaches 67.82 with Origin roughly unchanged. We do not downplay this point.
  • Coupling the positives still contributes on top of that. On the same data, a shared denominator beats the masked variant by about 6 points (73.90 vs. 67.82) and the independent-normalization variant by about 3 points (73.90 vs. 70.63), with Origin improving as well. This is consistent with the gradient analysis in Appendix E.

Response to W2 & Q2 & Q3 (View information differences; comparison and ablation settings)

Thank you for the review. We address three points: whether the generated descriptions preserve query-relevant information, whether differences in view information confound our modality-preference conclusion, and the comparison and ablation settings.

1. Generated descriptions and query-relevant information

The generated descriptions are not necessarily fully consistent with the images, and we do not claim the text and image views are strictly equivalent. Our main conclusions do not depend on their fidelity:

  • The distractor texts are not generated. The distractors in Fig. 3 are taken directly from a Wikipedia corpus. Irrelevant texts still cause a much larger NDCG@10 drop than the same number of irrelevant images (a Δ of 0.31 to 0.47 on ChartQA), so "Chaos in the Text" is not an artifact of generation.
  • Training and testing use different generators. We keep the model fixed (Trident is trained on text views from Qwen3-VL-235B) and replace only the test-corpus text views with versions generated by Qwen3.8-27B using the same pipeline.
ModelTest-corpus text fromMix Avg
Qwen3-VL-E-2BQwen3-VL-235B40.12
Qwen3-VL-E-2BQwen3.8-27B41.07
Trident-Qwen3VL-2BQwen3-VL-235B73.90
Trident-Qwen3VL-2BQwen3.8-27B74.00

Origin is a pure-image corpus and is unaffected. With the 27B text, Mix barely changes for either model, and the gap between them stays at roughly 33 points. If Trident had learned the 235B writing style, we would expect a clear drop. We do not observe one.

2. Do differences in view information confound the modality-preference conclusion?

The three views do carry different amounts of information: the text view is a detailed description, the image may contain evidence the description misses, and the fused view is a summary plus the original image (see also Appendix G). The higher similarity of text positives in Fig. 4(a) may therefore partly reflect information content. Our conclusion does not rest on that figure alone:

  • Irrelevant-document experiments (Fig. 3, Appendix B). Distractor texts and images are equally irrelevant to every query, so "text carries more information" cannot explain why irrelevant texts hurt much more.
  • Hard-negative interference (Fig. 4b). When the relevant document is an image, an irrelevant text hard negative outscores the positive in 76.0%–94.6% of cases. When the relevant document is text, this happens in only 15.8%–25.8%.
  • Before vs. after training (Fig. 7, Fig. 12). The views and their information content are identical before and after training, yet afterward the positive similarities of the three views converge (0.674 / 0.672 / 0.670) and interference rates fall to about 10–12%. If the gap came purely from information content, the training objective could not change it.
  • Forcing equality is not better (Appendix G). A Modality Balance Loss that explicitly equalizes the three view similarities brings no gain. Trident constrains only that the positive probability mass is not monopolized by one modality.

3. Comparison and ablation settings (Q3)

Trident is initialized from Qwen3-VL-2B-Instruct (Table 2), whereas Qwen3-VL-Embedding-2B is an officially trained model with different data and pipeline, so comparing the two directly is not a controlled comparison. We therefore added controls with the same initialization, training data (OpenDocVQA, 41k), and hyperparameters, changing only the listed factors (Avg is the mean over the six benchmarks in Table 1).

SettingPositivesNormalizationOrigin AvgMix Avg
Qwen3-VL-E-2B--77.5940.12
Control 1 (= "w/o Multi-Positive View InfoNCE" in Table 1)One random view per queryStandard InfoNCE77.3666.36
Control 2Three viewsPer-modality independent79.7670.63
TridentThree viewsShared80.5173.90
  • Fine-tuning accounts for part of the gain. It raises Mix from 40.12 to 66.36 with Origin essentially unchanged, so it mainly adapts the model to the mixed corpus and does not show that modality preference is removed.
  • Three-view data helps. From control 1 to control 2, Mix rises by 4.27 and Origin by 2.40.
  • Shared normalization adds a further gain. On identical three-view data, switching from independent to shared normalization raises Mix by 3.27 and Origin by 0.75. Independent normalization is insensitive to a constant offset applied to all candidates within one modality (Eq. (3), Appendix E.5), so it cannot fix the score-baseline gap between modalities. Shared normalization can. The original ablation (control 1) mixed the data and objective effects; separated, about 4.3 of the +7.5 Mix gain comes from the data and about 3.3 from the objective.

Response to Question 4

Thank you for pointing this out. Your derivation is correct: with three positives sharing one softmax denominator, the total positive mass satisfies ρ ≤ 1, so the full objective in Eq. (4) is lower-bounded by log 3 ≈ 1.099.

The curve in Figure 15(a) is not the full Eq. (4). It plots the −log ρ term, i.e., the relevance-discrimination term in the decomposition of Eq. (6):

L_trident = −log ρ + D_KL(u‖r) + log M

This term has a lower bound of 0, so a curve ending near 0, below log 3, is consistent with the equation. We will state this explicitly in the caption of Figure 15(a) and in the text of the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.