acceptodds
Under review as a conference paper at ICLR 2027

From Explicit CoT to Relevance Guidance: Discriminative Latent Reasoning for Universal Multimodal Embedding

Abstract

Reasoning has shown promise for improving universal multimodal embeddings, through both explicit chain-of-thought (CoT) and CoT-supervised latent reasoning. However, explicit CoT may not reliably convey the discriminative information that embeddings rely on to preserve distinctions among inputs. We argue that such information can be acquired and internalized by reasoning without necessarily relying on explicit CoT. To realize this view, we introduce Discriminative Latent Reasoning for Universal Multimodal Embedding (DiLRE), which establishes a discriminative foundation for latent reasoning through contrastive supervision and internalizes complementary information through relevance guidance. We train a Discriminative Guidance Model to jointly assess query–target relevance, providing guidance that complements embedding similarity. At test time, this guidance is combined with the embedding's current retrieval distribution to form discriminative targets for latent refinement. An iterative loop of retrieval and latent refinement then updates the latent reasoning tokens to align the resulting embedding's retrieval distribution with these targets. Extensive experiments and ablation studies demonstrate the effectiveness of DiLRE across diverse multimodal embedding tasks and model scales.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal multimodal embedding, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5EjvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiLRE, a universal multimodal embedding method that uses relevance guidance to refine query-side latent tokens without explicit CoT. A multimodal guidance model scores query–candidate relevance, and contrastive pretraining provides retrieval-oriented supervision for the latent tokens. At test time, DiLRE combines the guidance probabilities with the current retrieval distribution to form a discriminative target, then iteratively updates only the query’s latent-token inputs while keeping model parameters and candidate embeddings fixed. Experiments on MMEB-V1/V2 show that DiLRE outperforms embedding and reasoning baselines and achieves state-of-the-art performance. The paper also presents ablations and quality-throughput analyses over candidate budgets and refinement rounds, which demonstrate the effectiveness of the individual components and a favorable overall quality-throughput trade-off.

Critical strengths
  • DiLRE offers a viable approach to multimodal embedding without explicit reasoning and achieves state-of-the-art results across multiple datasets.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • No experiment demonstrates that the latent tokens learn reasoning ability. During pretraining, their mean-pooled states receive the same contrastive retrieval objective as the final embedding. At inference, the model freezes its parameters and updates only the query’s latent-token inputs to match a retrieval target formed from retrieval scores and guidance probabilities. These objectives supervise retrieval discrimination and distribution alignment, not reasoning steps or reasoning ability. The results are therefore consistent with test-time soft-prompt tuning and do not establish reasoning. The paper should add experiments analyzing whether the learned latent tokens have reasoning capability, for example by decoding or visualizing them in text or visual form and assessing whether they expose task-relevant intermediate reasoning.
  • The necessity of test-time latent refinement is not fully isolated, although refinement is the paper’s central contribution. Table 5 compares DiLRE (67.5) with static reranking (66.9) and direct embedding updates (66.5), while Table 13 reports 65.8 for validation-tuned static fusion. However, these controls do not test whether similar gains come from multi-turn candidate-pool refinement and aggregation of similarity and relevance scores without updating latent tokens. Without latent refinement, the method would largely reduce to adding a relevance model to similarity-based retrieval and combining scores across retrieval turns, leaving the main contribution closer to an engineering pipeline than to the claimed latent-reasoning mechanism. The authors need to show that latent-token refinement itself is necessary and yields a substantial, robust gain beyond these components.
  • DiLRE uses a separately trained relevance model to score retrieved candidates and feeds its scores back over multiple rounds. Its inference can therefore also be viewed as a multi-round VLM reranking pipeline with query adaptation, yet the paper does not compare it with other trained VLM rerankers under matched candidate pools, backbones, and compute. In particular, Table 1 reports LamRA scores of 40.4/47.4, below the base VLM2Vec results of 47.0/52.3, but does not specify whether these results are from LamRA-Ret or LamRA-Rank or explain the gap. Controlled reranker comparisons and clarification of the LamRA evaluation setting are needed to establish DiLRE’s advantage over reranking methods.
  • Figure 2 indicates that DiLRE’s inference involves repeated retrieval and top-M candidate selection, guidance-model scoring of query–candidate pairs, and latent-token updates through backpropagation. Figure 4 reports a 3.3× throughput increase when reducing M from 32 to 4 on one H20 GPU at batch size 1, with a 0.7-point drop in overall MMEB-V2 score. Appendix D.3 states that the reported wall-clock throughput includes latent updates, but the paper does not clearly specify whether it also covers all retrieval rounds, candidate selection, and guidance-model forward passes. Given these additional operations, the reported throughput appears close to VLM2Vec 2.0, and the paper should clarify the timing protocol and explain this result. Without a clear accounting of the full inference pipeline, it is difficult to compare DiLRE’s inference efficiency with other methods or assess its end-to-end deployment cost.
Questions for authors
  • Please refer to the Weaknesses section for the questions and concerns.
11 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank you for your thoughtful review and constructive suggestions, and for recognizing our method's effectiveness and the paper's clarity. We have revised the manuscript accordingly and respond point by point below.

Weakness 1: Latent reasoning and interpretability

TL;DR

  • Prior work has established a latent reasoning paradigm in which added special or soft tokens support intermediate computation without decoding their states into explicit CoT [1, 2, 3, 4, 5, 6, 7, 8]. DiLRE conforms to this paradigm and introduces direct discriminative supervision for learning and refining latent tokens in universal multimodal embedding.
  • We find that direct discriminative supervision of latent tokens yields better retrieval performance than the evaluated text-decoding objectives.

1. DiLRE conforms to an established latent reasoning paradigm

Prior work has established latent reasoning through intermediate computation in continuous hidden states, including those of added special or soft tokens [1, 2, 6, 7, 8]. These states can support the task without being decoded into explicit, human-readable CoT [2, 3, 4, 5]. The latent reasoning paradigm concerns how intermediate computation is performed, while the supervision objective determines how it is learned. Discriminative supervision is therefore compatible with this paradigm.

DiLRE conforms to this paradigm by inserting learnable special tokens between the multimodal input and the final embedding token. Each latent token attends to the multimodal input and preceding latent tokens, while the final embedding token attends to their contextualized states. These tokens therefore provide input-dependent intermediate computation that contributes to the final embedding. Updating their input embeddings changes this computation, and the query embedding is then recomputed from the original multimodal input and updated tokens.

2. Direct discriminative supervision of latent tokens outperforms the evaluated text-decoding supervision

Table R1 (reproduced from Table 3 of the original submission)compares latent-token supervision before test-time refinement while retaining the final-embedding contrastive objective. Direct contrastive supervision outperforms both the evaluated text-decoding objectives and training without an auxiliary latent loss, supporting the value of directly supervising these intermediate states for universal multimodal embedding.

Table R1. Direct contrastive supervision improves retrieval over training without an auxiliary latent loss and the evaluated text-decoding objectives.

Latent-token supervisionImage (36)Video (18)Visual document (24)All (78)
No latent tokens67.845.461.860.8
No auxiliary latent loss67.444.861.060.2
Text decoding: UME-R1 [9] targets67.042.269.562.0
Text decoding: RIME [10] targets66.742.169.962.0
Text decoding: Embed-RL [11] targets67.746.167.162.5
Contrastive supervision69.146.371.264.5

References

[1] Sachin Goyal et al. Think before you speak: Training Language Models With Pause Tokens. ICLR, 2024.

[2] Yuchang Sun et al. Enhancing Latent Computation in Transformers with Latent Tokens. arXiv:2505.12629, 2025.

[3] Kelvin Li et al. Latent Implicit Visual Reasoning. arXiv:2512.21218, 2025.

[4] Yinhan He et al. Implicit Reasoning for Large Language Model-based Generative Recommendation. arXiv:2606.14142, 2026.

[5] Wengao Ye et al. Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization. arXiv:2510.04182, 2025.

[6] Shibo Hao et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM, 2025.

[7] Yige Xu et al. SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. ACL, 2025.

[8] Zhenyi Shen et al. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP, 2025.

[9] Zhibin Lan et al. UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings. arXiv:2511.00405, 2025.

[10] Peixi Wu et al. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings. arXiv:2604.22280, 2026.

[11] Haonan Jiang et al. Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings. arXiv:2602.13823, 2026.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Contribution of test-time latent refinement

TL;DR

  • DiLRE's latent-token refinement outperforms the compared test-time retrieval refinement methods (reranking or direct embedding updates), as supported by controlled ablations and comparisons with existing methods [1, 2, 3, 4].
  • DiLRE differs fundamentally from existing test-time retrieval refinement methods by refining the latent computation that produces the query embedding, rather than only adjusting candidate rankings or output embedding vectors.
  • DiLRE provides a unified framework for universal multimodal embedding, using discriminative supervision to learn and refine latent reasoning during both training and inference.

1. DiLRE outperforms the compared test-time retrieval refinement methods

With the same pretrained encoder and guidance model, latent refinement outperforms static reranking and direct embedding updates (Table 5 of the original submission). The reranking control retains both retrieval similarity and guidance scores, while direct embedding updates use the same discriminative target as latent refinement.

On MMEB-V2 with 2B backbones (Table R2), DiLRE also outperforms test-time retrieval refinement methods GQR [3], ReFIT [2], and TOUR [1], as well as both modes of UMER [4], a state-of-the-art test-time retrieval refinement method for UME. For a fair comparison, ReFIT and TOUR use the same guidance model used in our method as their reranker, while GQR uses LaME [5], a state-of-the-art UME method, as its complementary retriever. We have added these baselines to the main comparison table (Table 1) of the revised manuscript.

Table R2. Latent refinement outperforms the compared test-time retrieval refinement methods.

Method / configurationImage (36)Video (18)Visual document (24)All (78)
GQR [3]68.245.971.364.0
ReFIT [2]67.745.671.063.6
VLM2Vec-V2 + TOUR [1]67.846.570.563.7
UMER-R [4]68.544.071.863.9
UMER-H [4]70.445.073.665.5
DiLRE, static reranking71.248.274.666.9
DiLRE, direct embedding update (TOUR [1])70.948.173.766.5
DiLRE71.749.375.067.5

2. DiLRE refines latent computation rather than candidate rankings or output embedding vectors

DiLRE refines the latent computation that produces the embedding. It backpropagates the retrieval objective through the frozen encoder to update the latent-token input embeddings, then recomputes the query embedding from the original multimodal input and updated tokens.

Each latent token attends to the multimodal input and preceding latent tokens, while the final embedding token attends to their contextualized states. This input-dependent computation allows the final embedding to incorporate both the discriminative information learned during contrastive pretraining and the query-specific relevance information added through refinement. Figure 3 of the revised submission shows that latent refinement yields greater mean displacement along relevance directions than direct embedding updates for the analyzed queries.

3. DiLRE unifies discriminative learning and refinement of latent reasoning for universal multimodal embedding

DiLRE provides a unified framework in which contrastive pretraining establishes a discriminative anchor for latent reasoning, and query-specific relevance feedback refines the learned computation at test time.

The ablations in Table 5 of the original submission support the benefit of combining the two stages. Removing latent contrastive supervision weakens subsequent refinement, while relevance-guided refinement further improves the contrastively pretrained encoder.

References

[1] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.

[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.

[3] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.

[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

[5] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

AuthorsOpenReview ↗

Response to Weakness 3

Weakness 3: Comparison with trained VLM rerankers and the LamRA evaluation setting

TL;DR

  • Our work focuses on learning and refining discriminative latent reasoning for universal multimodal embedding, rather than developing a stronger standalone reranker.
  • DiLRE outperforms the compared test-time retrieval refinement methods (GQR [1], ReFIT [2], and TOUR [3]) in fair comparisons, and also surpasses UMER [4], a state-of-the-art test-time retrieval refinement method for UME.
  • The reported LamRA scores are for LamRA-Ret, not LamRA-Rank [5]. We additionally evaluate test-time retrieval refinement on VLM2Vec-V2 using the same guidance model as our method [3, 6].

1. DiLRE provides a discriminative latent reasoning framework for universal multimodal embedding

DiLRE learns and refines latent reasoning through discriminative supervision, with the guidance model supplying relevance feedback. Comparing different rerankers alone does not isolate the benefit of latent refinement. We therefore compare refinement methods in controlled ablations using the same pretrained encoder and guidance model, without introducing differences in reranker training or capacity.

2. DiLRE outperforms the compared test-time retrieval refinement methods in fair comparisons

With the same pretrained encoder and guidance model, latent refinement outperforms reranking and direct embedding updates (Table 5 of the original submission). The reranking control combines retrieval similarity with guidance scores, while direct embedding updates use the same discriminative target as latent refinement.

On MMEB-V2 with 2B backbones, DiLRE also outperforms the test-time retrieval refinement methods GQR [1], ReFIT [2], and TOUR [3], as well as both modes of UMER [4], a state-of-the-art test-time retrieval refinement method for UME (Table R2 in our response to Weakness 2). ReFIT and TOUR use the same guidance model used in our method as their reranker, while GQR uses LaME [7], a state-of-the-art UME method, as its complementary retriever. We have added these methods to the main comparison table (Table 1) of the revised manuscript.

3. The LamRA entries report LamRA-Ret results, and DiLRE outperforms VLM2Vec-V2 + TOUR

The 40.4 and 47.4 entries in Table 1 of the original submission are LamRA-Ret retrieval-encoder results [5], not LamRA-Rank results on VLM2Vec. Their gap from VLM2Vec therefore does not measure the effect of reranking. We have clarified the LamRA-Ret labels in Table 1 of the revised manuscript.

We additionally evaluate VLM2Vec-V2 + TOUR [3, 6], with TOUR using the same guidance model used in our method as its reranker. TOUR uses relevance feedback to update the query embedding directly. DiLRE outperforms this configuration on MMEB-V2 with 2B backbones (Table R3).

Table R3. DiLRE outperforms VLM2Vec-V2 + TOUR across all three modality averages and overall.

MethodImage (36)Video (18)Visual document (24)All (78)
VLM2Vec-V2 + TOUR [3]67.846.570.563.7
DiLRE71.749.375.067.5

References

[1] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.

[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.

[3] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.

[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

[5] Yikun Liu et al. LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant. CVPR, 2025.

[6] Rui Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv:2507.04590, 2025.

[7] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

Reviewer_2ksfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DiLRE, a latent "reasoning" based UME model. Instead of training latent CoT as a compressed representation of textual CoT as in PLUME, this paper trains the latent tokens with the same infoNCE loss, and performs iterative refinements at test time.

Critical strengths
  1. I think there are certain novelty regarding using iterative refinement as a way of reranking and test-time scaling method.
Critical weaknesses
  1. I'm not sure about the intuition of using a separate contrastive loss over the mean-pooled latent tokens. First, the additional CoT in reasoning-based UME models is to make the final embedding more discriminative, yet in this case you are separately optimizing two contrastive loss. Second, there have been works such as MetaEmbed [1] and TTE-Flash [2], which show that additional learnable tokens can enhance retrieval performance even without the additional contrastive loss.
  2. It's not entirely fair to compare the proposed method (which includes reranking) with other non-reranking-based methods. I think it would be better to include the unrefined numbers into the main tables.
  3. One of the core idea of gradient-based iterative test-time refinement is similar to the previous work [3]. Although it's on text retrieval and has no latent tokens, the process seems to be quite similar.
  4. The number of refinements seems to be dataset-dependent and requires manual tuning.
  5. Table 1 seems to be missing a few previous works such as RIME, Embed-RL, MMEmb-R1 etc., which are indeed cited.
  6. The effect of using contrastive loss to supervise latent CoT is not well-ablated. Table 3 shows comparisons against explicit CoT-UME methods, but no comparison against similar latent CoT methods like PLUME, LAME and TTE-Flash. Furthermore, all the baseline numbers in Table 3 are substantially lower than their officially reported numbers (e.g. Embed-RL 2B is 66.8, but in Table 3 it's 62.5).

[1] MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

[2] TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens

[3] Optimizing Test-Time Query Representations for Dense Retrieval

Questions for authors

Please see weaknesses

12 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank you for your thoughtful review and for recognizing our method's novelty and the paper's clarity. We have revised the manuscript accordingly and respond point by point below.

Weakness 1: Contrastive supervision of latent tokens

TL;DR

  • Contrastive supervision for latent reasoning supplies discriminative information to the final embedding and establishes a discriminative anchor for refinement.
  • Contrastive supervision for latent reasoning achieves the best retrieval performance among the evaluated supervision strategies, without CoT annotation generation or auxiliary text decoding.
  • DiLRE uses contrastive supervision instead of explicit CoT (TTE-Flash [1]) and intermediate latent computation for single-vector retrieval, rather than multiple retrieval vectors (MetaEmbed [2]).

1. Latent supervision supports the final embedding and subsequent refinement

DiLRE uses latent tokens for latent reasoning through intermediate computation before forming the final embedding. Prior work trains such tokens with specific supervision, such as explicit CoT supervision in TTE-Flash [1]. We instead use direct contrastive supervision for latent reasoning tokens, so the final embedding token obtains discriminative information by attending to these latent states.

Contrastive pretraining also provides the learned latent-token input embeddings that serve as a discriminative anchor for relevance-guided refinement at test time. Removing latent contrastive supervision reduces performance after refinement (Table 5 of the original submission).

2. Contrastive supervision improves retrieval and simplifies training

Table R1 (Table 3 of the original submission) compares supervision for DiLRE's latent tokens before refinement, retaining the final-embedding contrastive objective. Contrastive supervision learns directly from retrieval pairs, removing CoT annotation generation and auxiliary text-decoding objectives. Even without refinement, DiLRE also outperforms the compared latent reasoning methods overall (Table R2).

Table R1. Direct contrastive supervision outperforms the evaluated text-decoding objectives and training without an auxiliary latent loss.

Latent-token supervisionImage (36)Video (18)Visual document (24)All (78)
No latent tokens67.845.461.860.8
No auxiliary latent loss67.444.861.060.2
Text decoding: UME-R1 [3] targets67.042.269.562.0
Text decoding: RIME [4] targets66.742.169.962.0
Text decoding: Embed-RL [5] targets67.746.167.162.5
Contrastive supervision69.146.371.264.5

Table R2. DiLRE outperforms the compared latent reasoning methods overall, even without refinement.

MethodImage (36)Video (18)Visual document (24)All (78)
TTE-Flash [1]68.350.668.164.1
LaME [6]69.344.572.164.4
PLUME [7]66.344.167.561.6
DiLRE (w/o refinement)69.146.371.264.5
DiLRE (with refinement)71.749.375.067.5

3. DiLRE uses contrastively supervised latent computation for single-vector retrieval

TTE-Flash [1] supervises its latent tokens with an explicit CoT-generation loss. DiLRE uses contrastive supervision instead, improving overall retrieval performance (Table R2) while simplifying training and removing dependence on CoT annotations.

MetaEmbed [2] uses all its added token states as multiple retrieval vectors, supervised through the contrastive retrieval objective. DiLRE uses latent tokens for intermediate computation and produces one retrieval vector. It outperforms MetaEmbed overall on MMEB-V1 while using fewer vectors per candidate (Table R3).

Table R3. DiLRE outperforms MetaEmbed while using one vector per candidate.

MethodVectors per candidateAll (36)
MetaEmbed-3B [2]6469.1
DiLRE-2B171.7

References

[1] Jianpeng Cheng et al. TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens. arXiv:2605.16638, 2026.

[2] Zilin Xiao et al. MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. ICLR, 2026.

[3] Zhibin Lan et al. UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings. arXiv:2511.00405, 2025.

[4] Peixi Wu et al. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings. arXiv:2604.22280, 2026.

[5] Haonan Jiang et al. Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings. arXiv:2602.13823, 2026.

[6] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

[7] Chenwei He et al. PLUME: Latent Reasoning Based Universal Multimodal Embedding. arXiv:2604.02073, 2026.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Comparison with non-reranking methods

TL;DR

  • We thank you for this suggestion and have added DiLRE (w/o refinement) to Table 1 of the revised manuscript. Even without test-time latent refinement, DiLRE achieves state-of-the-art performance in universal multimodal embedding (UME).
  • DiLRE outperforms static reranking with the same pretrained encoder and guidance model, and also surpasses both modes of UMER [1], a state-of-the-art test-time retrieval refinement method for UME.
  • DiLRE provides a unified framework for learning and refining latent reasoning through discriminative supervision, rather than only a test-time retrieval refinement method. Even considering test-time refinement alone, DiLRE differs fundamentally from existing methods by refining latent computation rather than only adjusting candidate rankings or the output embedding vector.

1. DiLRE without refinement is now included in the main table and outperforms the compared latent reasoning methods

We have added DiLRE (w/o refinement) to Table 1 of the revised manuscript. Using contrastive pretraining alone, this configuration outperforms the compared latent reasoning methods overall on MMEB-V2 with a 2B backbone (Table R2 in our response to Weakness 1).

2. DiLRE outperforms the compared test-time retrieval refinement methods (including reranking)

Latent refinement outperforms static reranking with the same pretrained encoder and guidance model (Table 5 of the original submission).

On MMEB-V2 with 2B backbones, DiLRE outperforms both modes of UMER [1], a state-of-the-art test-time retrieval refinement method for UME (Table R4). We have added these methods to the main comparison table (Table 1) of the revised manuscript.

Table R4. DiLRE outperforms both ranking and hybrid modes of UMER.

MethodImage (36)Video (18)Visual document (24)All (78)
UMER-R [1]68.544.071.863.9
UMER-H [1]70.445.073.665.5
DiLRE71.749.375.067.5

3. A unified framework for learning and refining latent reasoning for universal multimodal embedding

DiLRE provides a unified framework that learns and refines latent reasoning through discriminative supervision during both training and inference. Contrastive pretraining establishes the discriminative anchor, and relevance feedback further refines the learned latent computation for each query.

Even setting aside the pretraining stage, its test-time refinement optimizes the latent computation that produces the query embedding, rather than only adjusting candidate rankings or the output embedding vector.

References

[1] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

AuthorsOpenReview ↗

Response to Weakness 3

Weakness 3: Comparison with test-time query representation optimization

TL;DR

  • Even setting aside our contribution to latent reasoning supervision during pretraining, DiLRE differs fundamentally from existing test-time retrieval refinement methods. It refines the latent computation that produces the query embedding rather than directly optimizing the output embedding vector.
  • DiLRE outperforms TOUR [1] in a fair comparison, supporting the effectiveness of refining latent computation over directly optimizing the output query embedding.
  • DiLRE provides a unified latent reasoning framework for embedding, rather than only a test-time retrieval refinement method. It internalizes discriminative information into latent reasoning during both training and inference.

1. DiLRE refines latent computation rather than the output query embedding

Although both methods use gradient-based updates, DiLRE refines the latent computation that produces the embedding, whereas TOUR [1] directly optimizes the output query embedding. DiLRE backpropagates the retrieval objective through the frozen encoder to update the latent-token input embeddings, then recomputes the query embedding from the original multimodal input and the updated tokens.

Latent refinement acts through input-dependent contextualized states, so the embedding update is shaped by the encoder's learned computation over the original multimodal query. Figure 3 of the revised submission visualizes this effect, showing greater mean displacement along relevance directions than direct embedding updates for the analyzed queries.

2. DiLRE outperforms TOUR in a fair comparison

On MMEB-V2 with 2B backbones, DiLRE outperforms TOUR [1] when both use the same contrastively pretrained encoder, guidance model, and discriminative target (Table R5). We use TOUR's publicly released implementation [2]; this comparison corresponds to the direct embedding-update control in Table 5 of the original submission. DiLRE also outperforms TOUR applied to VLM2Vec-V2 [3].

Table R5. DiLRE outperforms TOUR applied to either VLM2Vec-V2 or DiLRE's contrastively pretrained encoder.

MethodImage (36)Video (18)Visual document (24)All (78)
VLM2Vec-V2 + TOUR [1]67.846.570.563.7
DiLRE (w/o refinement)69.146.371.264.5
DiLRE, contrastive pretraining + TOUR70.948.173.766.5
DiLRE71.749.375.067.5

3. DiLRE unifies discriminative learning and refinement of latent reasoning for universal multimodal embedding

DiLRE provides a unified framework that uses discriminative information both to learn latent reasoning through contrastive pretraining and to refine the learned latent computation using retrieval feedback at test time. Contrastive pretraining establishes a discriminative anchor, and query-specific relevance feedback further refines this learned computation.

The ablations in Table 5 of the original submission support the benefit of combining the two stages. Removing latent contrastive supervision weakens subsequent refinement, while relevance-guided refinement further improves the contrastively pretrained encoder.

References

[1] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.

[2] https://github.com/dmis-lab/TouR

[3] Rui Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv:2507.04590, 2025.

Reviewer_wdaaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims at improving universal multimodal embeddings through discriminative latent reasoning, without relying on explicit CoT supervision. The main technical contributions are contrastive latent supervision and relevance-guided test-time refinement. The former applies contrastive supervision to the mean-pooled states of a few latent reasoning tokens, establishing a discriminative anchor without CoT supervision. The latter trains a separate query–candidate scoring model whose relevance probabilities are combined with the query's current retrieval distribution at test time, iteratively refining the latent token embeddings with all model parameters and candidate embeddings frozen. The method is evaluated on MMEB-V1 and MMEB-V2 benchmark and shows consistent and broad performance advantages.

Critical strengths

This paper shows consistent and broad performance advantages across benchmarks. DiLRE outperforms latent reasoning methods on both MMEB-V1 and MMEB-V2, with consistent performance advantages across image, video, and visual-document tasks.

Critical weaknesses
  1. Limited gains from training and high inference costs.

DiLRE's main performance improvements come from additional computation at inference time, while its performance after training alone is roughly on par with LaME. Figure 4 shows that the default configuration achieves a throughput of only approximately 0.4 samples/s. DiLRE is even less efficient than the CoT-based method RIME. Given its low inference efficiency, DiLRE should be compared with CoT-based methods in terms of performance and inference time. It would be more convincing if stronger CoT-based methods are compared such as Think Then Embed [1].

  1. Missing comparisons with methods that optimize query representations using retrieval results at inference time.

Many existing methods optimize query representations using retrieval results at inference time. For example, GQR [2] uses candidate scores from a complementary retriever to construct a consensus distribution and iteratively optimizes the query representation by minimizing KL divergence. ReFIT [3] updates the query vector by distilling relevance scores from a reranker. The paper does not compare against such methods and therefore does not establish that iteratively refining latent embeddings is inherently superior to directly refining the retrieval query representation.

[1] Think Then Embed: Generative Context Improves Multimodal Embedding. ICLR 2026.

[2] Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR 2026.

[3] A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR 2025.

Questions for authors

My main concerns are the missing comparisons as listed in the weakness part. If additional evidence is provided and the proposed method demonstrates advantages over existing methods, I would increase my score:

  1. How does DiLRE compare with Think Then Embed in terms of performance and inference time?

  2. How does DiLRE's inference-time optimization method perform compared with methods such as GQR and ReFIT that directly optimize the retrieval query representation?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank you for your thoughtful review and constructive suggestions, and for recognizing the effectiveness of our approach and the clarity of our presentation. We have also revised the manuscript in response to your suggestions. Below, we provide our point-by-point responses.

Weakness 1: Training effectiveness and inference cost

  • Contrastive pretraining alone achieves state-of-the-art performance without explicit CoT supervision, slightly exceeding LaME [1]. Supervising both the final embedding and latent reasoning states directly from retrieval pairs removes the need for CoT annotation generation and auxiliary text decoding. The pretrained latent-token input embeddings also provide a discriminative anchor for test-time refinement, supported by the performance drop when latent contrastive supervision is removed (Table 5 of the original submission).
  • DiLRE offers a flexible performance–efficiency trade-off by adjusting the candidate budget MM and refinement rounds LL (Figure 4 of the original submission). Its low-latency configuration (M=4,L=1M=4,L=1) retains state-of-the-art performance, with throughput substantially higher than explicit reasoning methods (TTE [3] and RIME [4]).

All comparisons below use MMEB-V2 with 2B backbones. Table R1 reports throughput for a single H20 GPU at batch size 1. The baseline comparisons have also been included in the revised manuscript.

Table R1. DiLRE outperforms the compared baselines, while its low-latency configuration matches PLUME's throughput.

MethodReasoning typeMMEB-V2 score ↑Throughput (samples/s) ↑
TTE$_s$ [3]Explicit63.10.2
RIME [4]Explicit64.10.8
PLUME [2]Latent61.63.3
DiLRE (w/o refinement)Latent64.56.0
DiLRE (M=32,L≤5M=32,L\leq5)Latent67.50.4
DiLRE (M=4,L=1M=4,L=1)Latent66.43.3

Table R2. Both the high-performance and low-latency configurations of DiLRE outperform the compared baselines in overall retrieval performance.

MethodImage (36)Video (18)Visual document (24)All (78)
TTE$_s$ [3]70.132.168.863.1
TTE$_u$ [3]70.141.773.963.3
RIME [4]69.143.771.464.1
PLUME [2]66.344.167.561.6
DiLRE, contrastive pretraining only69.146.371.264.5
DiLRE, high performance (M=32,L≤5M=32, L\leq 5)71.749.375.067.5
DiLRE, low latency (M=4,L=1M=4, L=1)71.048.073.166.4

References

[1] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

[2] Chenwei He et al. PLUME: Latent Reasoning Based Universal Multimodal Embedding. arXiv:2604.02073, 2026.

[3] Xuanming Cui et al. Think Then Embed: Generative Context Improves Multimodal Embedding. ICLR, 2026.

[4] Peixi Wu et al. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings. arXiv:2604.22280, 2026.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Missing comparisons with methods that optimize query representations using retrieval results at inference time

TL;DR

  • DiLRE outperforms not only the compared test-time retrieval refinement methods[1][2] but also the state-of-the-art test-time retrieval refinement method specifically for Uiversal Multimodal Embedding (UME) [3].
  • DiLRE differs fundamentally from test-time query embedding optimization by refining the latent computation that produces the query embedding rather than directly optimizing the output embedding vector.
  • DiLRE provides a unified framework for learning and refining latent reasoning through discriminative supervision, rather than only a test-time retrieval refinement method.

1. DiLRE outperforms the compared test-time retrieval refinement methods

DiLRE outperforms GQR [1] and ReFIT [2] in fair comparisons on MMEB-V2 with 2B backbones using their publicly released implementations [4, 5]. ReFIT uses the same guidance model as DiLRE for reranking, while GQR uses LaME [6], a state-of-the-art UME method, as its complementary retriever.

DiLRE also outperforms both modes of UMER [3], a state-of-the-art test-time retrieval refinement method for UME. We have added these methods to the main comparison table (Table 1) of the revised manuscript.

Table R3. DiLRE outperforms GQR, ReFIT, and both UMER variants in overall retrieval performance and across all three modality averages.

MethodImage (36)Video (18)Visual document (24)All (78)
GQR [1]68.245.971.364.0
ReFIT [2]67.745.671.063.6
UMER-R [3]68.544.071.863.9
UMER-H [3]70.445.073.665.5
DiLRE (Ours)71.749.375.067.5

2. Refining latent computation rather than the final embedding vector

DiLRE refines latent computation, whereas test-time query embedding optimization methods [1, 2] directly update the output vector. It backpropagates the retrieval objective through the frozen encoder to update latent-token input embeddings, then recomputes the embedding from the original multimodal query. Relevance feedback thus changes input-dependent intermediate representations before the final embedding is formed.

The performance advantage is supported by Table 5 of the original submission. Figure 3 of the revised submission shows that latent refinement produces greater mean displacement along relevance directions than direct embedding updates for the analyzed queries.

3. A unified framework for learning and refining latent reasoning for universal multimodal embedding

The unified framework uses discriminative information both to learn latent reasoning through contrastive pretraining and to refine the learned latent computation using retrieval feedback at test time. During training, DiLRE applies contrastive supervision to the final embedding and latent reasoning states directly from retrieval pairs, without explicit CoT supervision or auxiliary text decoding. The learned latent-token input embeddings provide a discriminative anchor for inference-time refinement. At test time, the Discriminative Guidance Model supplies relevance feedback from retrieved candidates to further refine these tokens.

Our experimental results further demonstrate the benefit of combining the two stages (Table 5 of the original submission). Contrastive pretraining establishes the discriminative anchor for subsequent refinement: without latent contrastive supervision, the refined score drops from 67.5 to 64.8. Building on this anchor, relevance-guided refinement incorporates additional discriminative information into latent reasoning and raises the pretrained encoder's score from 64.5 to 67.5.

References

[1] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.

[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.

[3] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

[4] https://github.com/IBM/test-time-hybrid-retrieval

[5] https://github.com/gangiswag/llm-reranker#4-relevance-feedback

[6] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

AuthorsOpenReview ↗

Response to Questions

Question 1: Performance and inference time compared with Think Then Embed

DiLRE outperforms TTE [1] in overall retrieval performance, even without test-time latent refinement (Table R2).

The low-latency configurations further outperforms TTE [1], while the low-latency configuration achieves substantially higher throughput than that reported for TTE (Table R1).

 

Question 2: Comparison with inference-time optimization method

DiLRE outperforms existing test-time retrieval refinement methods (GQR [2] and ReFIT [3]) in fair comparisons (Table R3). DiLRE also outperforms both ranking and hybrid modes of UMER [4], a state-of-the-art test-time retrieval refinement method specifically for UME.

 

References

[1] Xuanming Cui et al. Think Then Embed: Generative Context Improves Multimodal Embedding. ICLR, 2026.

[2] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.

[3] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.

[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Shared Reviewer Concerns: DiLRE and Test-Time Retrieval Refinement

Dear Reviewers,

We sincerely thank you all for your thoughtful reviews and constructive suggestions. We appreciate the positive assessments of our approach's novelty and effectiveness, and the consistent recognition of the clarity of our presentation. We have provided point-by-point responses to your individual concerns.

All three reviews raise a shared question about DiLRE's relationship to existing test-time retrieval refinement methods. Although the original submission included relevant ablations, we provide a dedicated response to this concern below and have clarified the distinction in the revised manuscript. If our responses and the additional evidence address your concerns, we would be grateful if you could reconsider your assessments and ratings.

1. DiLRE refines latent computation rather than candidate rankings or output embedding vectors

Even setting aside our pretraining contribution and the unified framework, DiLRE differs fundamentally from existing test-time retrieval refinement methods. It refines the latent computation that produces the query embedding, rather than only adjusting candidate rankings or the output embedding vector.

DiLRE backpropagates the retrieval objective through the frozen encoder to update latent-token input embeddings, then recomputes the query embedding from the original multimodal input and updated tokens. Each latent token attends to the input and preceding latent tokens, while the final embedding token attends to their contextualized states. Relevance feedback therefore changes input-dependent intermediate computation before the final embedding is formed.

2. DiLRE unifies discriminative learning and refinement of latent reasoning for universal multimodal embedding

DiLRE provides a unified framework that uses discriminative information both to learn latent reasoning through contrastive pretraining and to refine the learned computation using retrieval feedback at test time.

Contrastive pretraining provides the learned latent-token input embeddings that serve as a discriminative anchor for relevance-guided refinement. The ablations in Table 5 of the original submission support the benefit of combining the two stages. Removing latent contrastive supervision weakens subsequent refinement, while refinement further improves the contrastively pretrained encoder.

3. DiLRE outperforms the compared test-time retrieval refinement methods

DiLRE outperforms GQR [1], ReFIT [2], and TOUR [3], as well as both modes of UMER [4], a state-of-the-art test-time retrieval refinement method for UME. Table R1 summarizes the MMEB-V2 results with 2B backbones. ReFIT and TOUR use the same guidance model used in our method as their reranker, while GQR uses LaME [5] as its complementary retriever.

With the same pretrained encoder and guidance model, latent refinement also outperforms static reranking and direct embedding updates in the controlled ablations (Table 5 of the original submission).

Table R1. DiLRE outperforms the compared test-time retrieval refinement methods overall and across all three modality averages.

MethodImage (36)Video (18)Visual document (24)All (78)
GQR [1]68.245.971.364.0
ReFIT [2]67.745.671.063.6
TOUR [3]67.846.570.563.7
UMER-R [4]68.544.071.863.9
UMER-H [4]70.445.073.665.5
DiLRE71.749.375.067.5

Revisions to the manuscript

  • Related work: Added a discussion of test-time retrieval refinement, including reranking and direct embedding optimization, to clarify the distinction from DiLRE.
  • Main comparisons: Added comparisons with baselines suggested by the reviewers and additional relevant test-time retrieval refinement methods to the main comparison table (Table 1).
  • Visual analysis: Added Figure 3, which shows greater mean displacement along relevance directions under latent refinement than under direct embedding updates for the analyzed queries.

References

[1] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.

[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.

[3] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.

[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.

[5] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.