- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DiLRE, a universal multimodal embedding method that uses relevance guidance to refine query-side latent tokens without explicit CoT. A multimodal guidance model scores query–candidate relevance, and contrastive pretraining provides retrieval-oriented supervision for the latent tokens. At test time, DiLRE combines the guidance probabilities with the current retrieval distribution to form a discriminative target, then iteratively updates only the query’s latent-token inputs while keeping model parameters and candidate embeddings fixed. Experiments on MMEB-V1/V2 show that DiLRE outperforms embedding and reasoning baselines and achieves state-of-the-art performance. The paper also presents ablations and quality-throughput analyses over candidate budgets and refinement rounds, which demonstrate the effectiveness of the individual components and a favorable overall quality-throughput trade-off.
Critical strengths
- DiLRE offers a viable approach to multimodal embedding without explicit reasoning and achieves state-of-the-art results across multiple datasets.
- The paper is clearly written and easy to follow.
Critical weaknesses
- No experiment demonstrates that the latent tokens learn reasoning ability. During pretraining, their mean-pooled states receive the same contrastive retrieval objective as the final embedding. At inference, the model freezes its parameters and updates only the query’s latent-token inputs to match a retrieval target formed from retrieval scores and guidance probabilities. These objectives supervise retrieval discrimination and distribution alignment, not reasoning steps or reasoning ability. The results are therefore consistent with test-time soft-prompt tuning and do not establish reasoning. The paper should add experiments analyzing whether the learned latent tokens have reasoning capability, for example by decoding or visualizing them in text or visual form and assessing whether they expose task-relevant intermediate reasoning.
- The necessity of test-time latent refinement is not fully isolated, although refinement is the paper’s central contribution. Table 5 compares DiLRE (67.5) with static reranking (66.9) and direct embedding updates (66.5), while Table 13 reports 65.8 for validation-tuned static fusion. However, these controls do not test whether similar gains come from multi-turn candidate-pool refinement and aggregation of similarity and relevance scores without updating latent tokens. Without latent refinement, the method would largely reduce to adding a relevance model to similarity-based retrieval and combining scores across retrieval turns, leaving the main contribution closer to an engineering pipeline than to the claimed latent-reasoning mechanism. The authors need to show that latent-token refinement itself is necessary and yields a substantial, robust gain beyond these components.
- DiLRE uses a separately trained relevance model to score retrieved candidates and feeds its scores back over multiple rounds. Its inference can therefore also be viewed as a multi-round VLM reranking pipeline with query adaptation, yet the paper does not compare it with other trained VLM rerankers under matched candidate pools, backbones, and compute. In particular, Table 1 reports LamRA scores of 40.4/47.4, below the base VLM2Vec results of 47.0/52.3, but does not specify whether these results are from LamRA-Ret or LamRA-Rank or explain the gap. Controlled reranker comparisons and clarification of the LamRA evaluation setting are needed to establish DiLRE’s advantage over reranking methods.
- Figure 2 indicates that DiLRE’s inference involves repeated retrieval and top-M candidate selection, guidance-model scoring of query–candidate pairs, and latent-token updates through backpropagation. Figure 4 reports a 3.3× throughput increase when reducing M from 32 to 4 on one H20 GPU at batch size 1, with a 0.7-point drop in overall MMEB-V2 score. Appendix D.3 states that the reported wall-clock throughput includes latent updates, but the paper does not clearly specify whether it also covers all retrieval rounds, candidate selection, and guidance-model forward passes. Given these additional operations, the reported throughput appears close to VLM2Vec 2.0, and the paper should clarify the timing protocol and explain this result. Without a clear accounting of the full inference pipeline, it is difficult to compare DiLRE’s inference efficiency with other methods or assess its end-to-end deployment cost.
Questions for authors
- Please refer to the Weaknesses section for the questions and concerns.
11 comments
Response to Weakness 2
Weakness 2: Contribution of test-time latent refinement
TL;DR
- DiLRE's latent-token refinement outperforms the compared test-time retrieval refinement methods (reranking or direct embedding updates), as supported by controlled ablations and comparisons with existing methods [1, 2, 3, 4].
- DiLRE differs fundamentally from existing test-time retrieval refinement methods by refining the latent computation that produces the query embedding, rather than only adjusting candidate rankings or output embedding vectors.
- DiLRE provides a unified framework for universal multimodal embedding, using discriminative supervision to learn and refine latent reasoning during both training and inference.
1. DiLRE outperforms the compared test-time retrieval refinement methods
With the same pretrained encoder and guidance model, latent refinement outperforms static reranking and direct embedding updates (Table 5 of the original submission). The reranking control retains both retrieval similarity and guidance scores, while direct embedding updates use the same discriminative target as latent refinement.
On MMEB-V2 with 2B backbones (Table R2), DiLRE also outperforms test-time retrieval refinement methods GQR [3], ReFIT [2], and TOUR [1], as well as both modes of UMER [4], a state-of-the-art test-time retrieval refinement method for UME. For a fair comparison, ReFIT and TOUR use the same guidance model used in our method as their reranker, while GQR uses LaME [5], a state-of-the-art UME method, as its complementary retriever. We have added these baselines to the main comparison table (Table 1) of the revised manuscript.
Table R2. Latent refinement outperforms the compared test-time retrieval refinement methods.
| Method / configuration | Image (36) | Video (18) | Visual document (24) | All (78) |
|---|---|---|---|---|
| GQR [3] | 68.2 | 45.9 | 71.3 | 64.0 |
| ReFIT [2] | 67.7 | 45.6 | 71.0 | 63.6 |
| VLM2Vec-V2 + TOUR [1] | 67.8 | 46.5 | 70.5 | 63.7 |
| UMER-R [4] | 68.5 | 44.0 | 71.8 | 63.9 |
| UMER-H [4] | 70.4 | 45.0 | 73.6 | 65.5 |
| DiLRE, static reranking | 71.2 | 48.2 | 74.6 | 66.9 |
| DiLRE, direct embedding update (TOUR [1]) | 70.9 | 48.1 | 73.7 | 66.5 |
| DiLRE | 71.7 | 49.3 | 75.0 | 67.5 |
2. DiLRE refines latent computation rather than candidate rankings or output embedding vectors
DiLRE refines the latent computation that produces the embedding. It backpropagates the retrieval objective through the frozen encoder to update the latent-token input embeddings, then recomputes the query embedding from the original multimodal input and updated tokens.
Each latent token attends to the multimodal input and preceding latent tokens, while the final embedding token attends to their contextualized states. This input-dependent computation allows the final embedding to incorporate both the discriminative information learned during contrastive pretraining and the query-specific relevance information added through refinement. Figure 3 of the revised submission shows that latent refinement yields greater mean displacement along relevance directions than direct embedding updates for the analyzed queries.
3. DiLRE unifies discriminative learning and refinement of latent reasoning for universal multimodal embedding
DiLRE provides a unified framework in which contrastive pretraining establishes a discriminative anchor for latent reasoning, and query-specific relevance feedback refines the learned computation at test time.
The ablations in Table 5 of the original submission support the benefit of combining the two stages. Removing latent contrastive supervision weakens subsequent refinement, while relevance-guided refinement further improves the contrastively pretrained encoder.
References
[1] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.
[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.
[3] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.
[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.
[5] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.
Response to Weakness 3
Weakness 3: Comparison with trained VLM rerankers and the LamRA evaluation setting
TL;DR
- Our work focuses on learning and refining discriminative latent reasoning for universal multimodal embedding, rather than developing a stronger standalone reranker.
- DiLRE outperforms the compared test-time retrieval refinement methods (GQR [1], ReFIT [2], and TOUR [3]) in fair comparisons, and also surpasses UMER [4], a state-of-the-art test-time retrieval refinement method for UME.
- The reported LamRA scores are for LamRA-Ret, not LamRA-Rank [5]. We additionally evaluate test-time retrieval refinement on VLM2Vec-V2 using the same guidance model as our method [3, 6].
1. DiLRE provides a discriminative latent reasoning framework for universal multimodal embedding
DiLRE learns and refines latent reasoning through discriminative supervision, with the guidance model supplying relevance feedback. Comparing different rerankers alone does not isolate the benefit of latent refinement. We therefore compare refinement methods in controlled ablations using the same pretrained encoder and guidance model, without introducing differences in reranker training or capacity.
2. DiLRE outperforms the compared test-time retrieval refinement methods in fair comparisons
With the same pretrained encoder and guidance model, latent refinement outperforms reranking and direct embedding updates (Table 5 of the original submission). The reranking control combines retrieval similarity with guidance scores, while direct embedding updates use the same discriminative target as latent refinement.
On MMEB-V2 with 2B backbones, DiLRE also outperforms the test-time retrieval refinement methods GQR [1], ReFIT [2], and TOUR [3], as well as both modes of UMER [4], a state-of-the-art test-time retrieval refinement method for UME (Table R2 in our response to Weakness 2). ReFIT and TOUR use the same guidance model used in our method as their reranker, while GQR uses LaME [7], a state-of-the-art UME method, as its complementary retriever. We have added these methods to the main comparison table (Table 1) of the revised manuscript.
3. The LamRA entries report LamRA-Ret results, and DiLRE outperforms VLM2Vec-V2 + TOUR
The 40.4 and 47.4 entries in Table 1 of the original submission are LamRA-Ret retrieval-encoder results [5], not LamRA-Rank results on VLM2Vec. Their gap from VLM2Vec therefore does not measure the effect of reranking. We have clarified the LamRA-Ret labels in Table 1 of the revised manuscript.
We additionally evaluate VLM2Vec-V2 + TOUR [3, 6], with TOUR using the same guidance model used in our method as its reranker. TOUR uses relevance feedback to update the query embedding directly. DiLRE outperforms this configuration on MMEB-V2 with 2B backbones (Table R3).
Table R3. DiLRE outperforms VLM2Vec-V2 + TOUR across all three modality averages and overall.
| Method | Image (36) | Video (18) | Visual document (24) | All (78) |
|---|---|---|---|---|
| VLM2Vec-V2 + TOUR [3] | 67.8 | 46.5 | 70.5 | 63.7 |
| DiLRE | 71.7 | 49.3 | 75.0 | 67.5 |
References
[1] Omri Uzan et al. Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization. ICLR, 2026.
[2] Revanth Gangi Reddy et al. A Large-Scale Study of Reranker Relevance Feedback at Inference. SIGIR, 2025.
[3] Mujeen Sung et al. Optimizing Test-Time Query Representations for Dense Retrieval. Findings of ACL, 2023.
[4] Libiao Chen et al. UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval. arXiv:2608.18504, 2026.
[5] Yikun Liu et al. LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant. CVPR, 2025.
[6] Rui Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv:2507.04590, 2025.
[7] Peixi Wu et al. LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck. arXiv:2606.13061, 2026.
Response to Weakness 1
We thank you for your thoughtful review and constructive suggestions, and for recognizing our method's effectiveness and the paper's clarity. We have revised the manuscript accordingly and respond point by point below.
Weakness 1: Latent reasoning and interpretability
TL;DR
1. DiLRE conforms to an established latent reasoning paradigm
Prior work has established latent reasoning through intermediate computation in continuous hidden states, including those of added special or soft tokens [1, 2, 6, 7, 8]. These states can support the task without being decoded into explicit, human-readable CoT [2, 3, 4, 5]. The latent reasoning paradigm concerns how intermediate computation is performed, while the supervision objective determines how it is learned. Discriminative supervision is therefore compatible with this paradigm.
DiLRE conforms to this paradigm by inserting learnable special tokens between the multimodal input and the final embedding token. Each latent token attends to the multimodal input and preceding latent tokens, while the final embedding token attends to their contextualized states. These tokens therefore provide input-dependent intermediate computation that contributes to the final embedding. Updating their input embeddings changes this computation, and the query embedding is then recomputed from the original multimodal input and updated tokens.
2. Direct discriminative supervision of latent tokens outperforms the evaluated text-decoding supervision
Table R1 (reproduced from Table 3 of the original submission)compares latent-token supervision before test-time refinement while retaining the final-embedding contrastive objective. Direct contrastive supervision outperforms both the evaluated text-decoding objectives and training without an auxiliary latent loss, supporting the value of directly supervising these intermediate states for universal multimodal embedding.
Table R1. Direct contrastive supervision improves retrieval over training without an auxiliary latent loss and the evaluated text-decoding objectives.
References
[1] Sachin Goyal et al. Think before you speak: Training Language Models With Pause Tokens. ICLR, 2024.
[2] Yuchang Sun et al. Enhancing Latent Computation in Transformers with Latent Tokens. arXiv:2505.12629, 2025.
[3] Kelvin Li et al. Latent Implicit Visual Reasoning. arXiv:2512.21218, 2025.
[4] Yinhan He et al. Implicit Reasoning for Large Language Model-based Generative Recommendation. arXiv:2606.14142, 2026.
[5] Wengao Ye et al. Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization. arXiv:2510.04182, 2025.
[6] Shibo Hao et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM, 2025.
[7] Yige Xu et al. SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. ACL, 2025.
[8] Zhenyi Shen et al. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP, 2025.
[9] Zhibin Lan et al. UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings. arXiv:2511.00405, 2025.
[10] Peixi Wu et al. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings. arXiv:2604.22280, 2026.
[11] Haonan Jiang et al. Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings. arXiv:2602.13823, 2026.