acceptodds
Under review as a conference paper at ICLR 2027

MATE: Matching-Aware Latent Reasoning for Universal Multimodal Embedding

Abstract

Universal multimodal embedding (UME) enables scalable retrieval by encoding queries and candidates independently in a shared vector space. Complex retrieval can nevertheless benefit from integrating evidence within each input and directly comparing queries with semantically similar candidates. Explicit reasoning and multi-step latent rollout increase encoding cost, while pairwise matching adds per-candidate computation at inference. We propose MATE, a one-step bi-encoder that learns from both forms of computation. Latent Reasoning Distillation (LRD) uses answer-guided weighting to summarize a teacher's multi-step latent trajectory and progressively distills this target into a single latent step. Latent-Aware Matching Distillation (LAMD) trains a matcher on paired latent states and mean-pooled embeddings, then selectively transfers its listwise preferences to the bi-encoder. Matching distillation focuses on queries with a sufficient matcher margin but an insufficient bi-encoder margin. At inference, MATE encodes queries and candidates independently and ranks candidates by normalized vector similarity. On the 78-task MMEB-V2 benchmark, the 2B MATE model achieves an overall score of 65.4 and delivers 62 the throughput of the explicit-reasoning model UME-R1 under matched inference settings. Code will be publicly available upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal Multimodal Embedding, Latent Reasoning, Progressive Distillation, Efficient Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_22NDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MATE, which consists of Latent Reasoning Distillation (LRD) and Latent-Aware Matching Distillation (LAMD). LRD trains a multi-step latent teacher with explicit rationales and progressively distills its trajectory into a single latent state, while LAMD uses a training-only matcher to examine latent states and pooled embeddings of queries and candidates and transfers listwise matching preferences to a bi-encoder.

Critical strengths
  • LRD enables the teacher's multi-step reasoning traces to be distilled into a single latent state. In addition, LAMD uses a matcher only during training to align the latent state and embeddings of queries and candidates provide ranking supervision for the bi-encoder so that the candidates are ranked by embeddings of query and candidates at inference time.
  • The authors compared MATE with recent UME models across discriminative, explicit-reasoning, and latent-reasoning approaches on the 78 tasks included in MMEB-V2, and demonstrates improved average performance.
Critical weaknesses
  • The paper claims that the 2B MATE model achieves an overall score of 65.4 on the 78-task MMEB-V2 benchmark while delivering 62× higher throughput than UME-R1 under matched inference settings. Thus, a key contribution is not only improved effectiveness but also efficiency. However, the inference setup for the throughput comparison is not clearly described, making it difficult to assess the fairness and reproducibility. Additionally, the paper lacks further analysis of the throughput results beyond Figure 1.
  • There are several experimental results that are not sufficiently explained or discussed.
Questions for authors
  • The paper does not analyze the performance impact of discarding the matcher during inference. Reporting results with the matcher used at inference time would provide a useful upper bound and help quantify the trade-off between effectiveness and efficiency.

  • In Figure 1, the x-axis label appears to be misspelled as "Throughout". It should likely be "Throughput".

  • The MMEB-V2 results reported in Figure 1 do not appear to match the Avg scores in Table 1. It would be helpful to clarify how these results were obtained and why the values differ, so that readers can clearly understand whether the discrepancy arises from different evaluation settings.

  • Figure 2 is not discussed in the main text. In addition, to the best of my knowledge, the categorization and illustrations in Figure 2(a)-(c) are almost identical to those presented in LaME. Since these concepts do not appear to have been first introduced in this work, it would be helpful to explicitly acknowledge the prior work and clarify the relationship between this paper and LaME.

  • The largest gains in Table 1 are observed on GD in Image and MERT in video. Further analysis would help clarify when the proposed method is most effective. Table 2 includes a comparison between IND and OOD settings intentionally, but the results are not discussed in the paper.

1 comment
AuthorsOpenReview ↗

Inference Efficiency and Result Analysis

Thank you for recognizing MATE's effectiveness and practical benefits. Our single-step encoder retains the gains from reasoning and matching supervision during training. The new comparison below shows that retaining the matcher adds only 0.1 points while increasing latency by 37%.

1. Inference setup and throughput

We measure query-encoding throughput with batch size 1, BF16, FlashAttention-2, and no quantization. MATE and UME-R1 use the same hardware and inference settings. MATE achieves 6.24 ± 0.02 samples/s, compared with 0.10 ± 0.01 samples/s for UME-R1, corresponding to approximately 62× the throughput.

Timing includes image loading, preprocessing, and query embedding generation, with CUDA synchronization. It excludes model loading, warm-up, offline corpus encoding, search, and reranking. MATE uses one latent step and avoids generating a textual rationale before producing an embedding. Neither its teacher nor its matcher is needed at inference. Further implementation details are provided in Supplementary Section B.

2. What happens if the matcher is retained?

We compare the final Qwen2-VL-2B encoder with and without reranking. Both modes rank the original candidate pool by embedding similarity. The second then uses the matcher to rerank the top 16 candidates. The encoder was trained with four hard negatives. The encoder and matcher remain fixed, and the training selection rule is not used at inference.

Table A. Effect of retaining the matcher on MMEB-V2.

Inference modeImageVideoVisDocOverallLatency (ms/query)
Encoder only69.747.372.665.4183
Encoder + matcher69.747.472.765.5251

Both modes use the same queries, candidate pools, hardware, precision, and query batch size. Candidate representations are cached offline, and each query is encoded once. Latency runs from prepared inputs to the final ranking, including query encoding, candidate scoring, and reranking when enabled. Its scope differs from the throughput measurement above.

Reranking gains 0.1 points and adds 68 ms/query. In this top-16 setting, the default encoder achieves nearly the same score at lower latency. The paper's matching gain, from 64.0 to 65.4, already uses no matcher at inference.

3. Figure 1 and the relationship to LaME in Figure 2

Figure 1 was assembled manually, so some markers were not positioned precisely at their corresponding values. The underlying results are the 78-task overall scores in the All column of Table 1. We will correct the marker positions to match these values. The x-axis label has also been corrected to “Throughput”.

Figure 2 was independently designed to compare established retrieval approaches with MATE. Panels (a)–(c) summarize existing paradigms, while panel (d) illustrates our method. We discuss the figure in the main text and clarify the methodological differences between MATE and related work, including LaME. LaME trains latent tokens with reconstruction and retrieval objectives. MATE trains an eight-step latent teacher, compresses it to one step through LRD, and uses LAMD to transfer matching supervision into the encoder. Panel (d) will make this contribution clearer.

4. Grounding, moment retrieval, and IND/OOD results

At the 2B scale, MATE improves over LaME by 10.3 points on grounding and 7.4 points on moment retrieval . These tasks require identifying a particular object or temporal segment among candidates that share much of their visual content. Retaining relevant attributes, relations, and actions is therefore useful, as the examples in Supplementary §E illustrate.

The ablations support both training stages: answer-guided supervision raises the video score from 41.4 to 44.9 over final-state supervision, and matching distillation further raises it to 47.3. This is consistent with preserving useful content during compression and then improving discrimination between similar candidates.

On the image OOD split, MATE scores 67.2 vs. 64.4 for LaME; on IND, it scores 71.6 vs. 73.1. We will discuss both results explicitly. The gain over LaME on this image benchmark comes from the OOD tasks, while LaME remains stronger on the IND tasks.

We will add these details to clarify when MATE helps and how its inference cost is measured.

Reviewer_7c2xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MATE, a universal multimodal embedding framework that transfers supervision from a multi-step latent teacher and a query–candidate matcher into a one-step bi-encoder. The two main components are Latent Reasoning Distillation (LRD) and Latent-Aware Matching Distillation (LAMD). LRD distills an answer-guided summary of the teacher's latent trajectory. LAMD uses the matcher to provide listwise supervision for the bi-encoder. Moreover, the core designs show advantages on the MMEB V2 dataset.

Critical strengths

The paper addresses a real deployment problem for large-scale retrieval: transferring richer reasoning and matching signals into a bi-encoder while preserving independently indexable embeddings. Moreover, LAMD demonstrate advantages over the baselines on the MMEB V2 dataset in terms of performance and inference time.

Critical weaknesses
  1. The central distillation algorithm is not compared with established methods of the same kind.

The LRD evaluation in Table 5 compares the proposed answer-guided target only against two simple baselines: final-state distillation and uniform trajectory averaging. It does not compare LRD with representative latent-reasoning distillation methods, such as CODI[1], LaSER[2] and System-1.5 Reasoning[3]. The results therefore show an advantage over these two simple design choices, but do not establish an advantage over existing distillation algorithms. Comparisons between complete retrieval systems do not resolve this gap because they do not isolate LRD. Since LRD is a central methodological contribution, this missing comparison is the primary reason I cannot support acceptance.

  1. The benefit of LAMD’s selection rule is not clearly established.

LAMD checks the matcher and bi-encoder margins on the bi-encoder’s hardest negative, but then distills the matcher’s distribution over the entire candidate set. It is unclear whether this pairwise criterion identifies useful supervision for the remaining candidates. Table 4 supports the overall benefit of LAMD, but does not isolate the effect of this selection rule. Comparisons with other design choices or alternative selection criteria using the same matcher would help establish its value.

[1] CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP 2025

[2] LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval. SIGIR 26

[3] System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts. NeurIPS 2025.

Questions for authors
  1. How does LRD compare with representative latent distillation methods?

  2. Does LAMD's selection rule outperform other design choices using the same matcher?

5 comments
AuthorsOpenReview ↗

Distillation Baselines and Query Selection

Thank you for recognizing MATE's practical benefits. On MMEB-V2, progressive compression gives the one-step student a 4.5-point gain over direct compression. In a separate comparison, the encoder scores 1.4 points higher with matching supervision on selected queries than on all queries. We have marked the changes in blue in the revised paper and supplement.

1. Training data and what “answer” means in non-QA tasks

We train the teacher on the public UME-R1 supervised fine-tuning corpus. It already provides rationales and answers for both queries and candidates. Our training mix covers 22 image or document datasets and three video datasets. We use subset-balanced sampling and reuse the same training pairs across stages. The revised supplement lists the datasets and gives annotation examples (Sections B and C).

The retrieval label tells us which candidate is correct. The <answer> field is a short text summary. It can name a category in classification or describe an object in a matching image. In document retrieval, it can summarize a page. The page itself is what the model must retrieve. A query annotation can use a paired answer as context when one is available. Candidates are annotated from their own content alone.

To form the student's target, we weight the teacher's reasoning states by their similarity to the answer representation. The retrieval embedding comes from states that precede the answer text. At inference, MATE uses only the one-step encoder to embed each query or candidate.

2. Progressive versus direct compression

The “One-step baseline” in Table 3 of the original submission is our direct-compression control. Both students start from the same checkpoint. We keep the teacher and answer-guided target fixed and use the same data, losses, and number of student updates. Direct compression uses one latent step throughout Stage 2. With progressive Latent Reasoning Distillation (LRD), we use eight steps for 20% of the updates, four for 20%, two for 25%, and one for the final 35% (Supplementary Table 1).

Table A. Direct and progressive compression on MMEB-V2 before matching distillation (Qwen2-VL-2B). In both tables, Overall is the unweighted mean across all 78 tasks.

MethodInference stepsOverall
Teacher864.7
Direct compression159.5
Progressive LRD164.0

With one step at inference, the progressive student comes within 0.7 points of the eight-step teacher.

3. Does query selection help?

We have added a control that applies the matching loss to every query. All three runs in Table B start from the same LRD checkpoint. Each uses the contrastive retrieval loss (InfoNCE) and trains for the same number of updates.

Both distillation runs use the same frozen matcher, temperatures, and KL weight. The matcher scores one positive and four hard negatives per query. We average the KL loss over the full batch (Eq. 14). Unselected queries contribute only to InfoNCE.

Table B. Effect of query selection on MMEB-V2 (Qwen2-VL-2B).

Matching supervisionImageVideoVisual documentsOverall
None (InfoNCE only)68.245.171.563.9
All queries67.946.671.164.0
Selected queries69.747.372.665.4

Using the matcher on every query improves the video score but lowers the image and document scores compared with InfoNCE alone. The overall score changes little, from 63.9 to 64.0. Selecting queries improves all three averages over the all-query run and brings the overall score to 65.4.

4. Why these thresholds?

For each query, we compare the positive with the negative that the encoder scores highest among the four. We use matching supervision only when both conditions in Eq. 11 hold. The matcher must rank the positive strictly higher, so we set its threshold to zero (δ_M = 0). The encoder's margin (positive score minus negative score) must be below δ_E = 0.05. This includes errors, ties, and correct rankings with a small lead. A cutoff of zero would keep only errors. At the encoder's distillation temperature of 0.05, a cosine similarity gap of 0.05 corresponds to one logit unit (Eq. 13).

Thank you for providing the additional experiments, which offer further evidence of the proposed method’s advantages.

Could you provide additional empirical evidence or theoretical analysis to explain why the proposed method is superior to the compared approaches or alternative design choices?

AuthorsOpenReview ↗

Thank you for the follow-up.

We can make the reasoning behind these choices more precise through the training objectives.

1. Progressive compression

The answer-guided target zx∗=∑j=1Nαx,jh^x,j\mathbf{z}_x^*=\sum_{j=1}^{N}\alpha_{x,j}\hat{\mathbf{h}}_{x,j} in Eq. 5 combines teacher states using weights based on their similarity to the training answer representation. It provides one target for students with different numbers of steps. Using the notation of Eqs. 6 and 15,

\mathcal{L}_{\mathrm{LRD}}^{(k)} &=\operatorname{SmoothL1}\left(\mathbf{z}_x^{(k)},\operatorname{sg}(\mathbf{z}_x^*)\right),\\\\ \mathcal{L}^{(k)} &=\mathcal{L}_{\mathrm{con}}+\lambda_{\mathrm{LRD}}\mathcal{L}_{\mathrm{LRD}}^{(k)}. \end{aligned}$$ Here, $\mathbf{z}_x^{(k)}$ is the mean of the student's $k$ latent states, while $\mathbf{z}_x^*$ stays fixed as $k$ decreases. Direct and progressive compression use the same final objective $\mathcal{L}^{(1)}$ and the same one-step architecture. Progressive compression changes the training path: each shorter stage starts from parameters learned with more steps, while supervision still comes from the original teacher. The same final objective need not lead to the same result after a finite number of updates. The schedule is intended to help the student adapt to reduced computation in stages while preserving the reference it learns from. The direct-compression control in our previous reply tests this training choice under the same number of student updates; Table 5 separately tests the choice of the answer-guided target. **2. Query selection for matching distillation** For query $q_i$, let $c_i^+$ and $c_i^{\mathrm{hard}}$ be the positive and hardest negative from Eqs. 9-11. Write $p_i^{\mathcal{M}}(c)$ and $p_i^{\mathcal{E}}(c)$ for the probabilities assigned to candidate $c$ in Eqs. 12-13. Holding the matcher target and selection mask fixed, Eq. 14 gives the score derivative $$\frac{\partial\mathcal{L}_{\mathrm{LAMD}}} {\partial(\mathbf{e}_{q_i}^{\top}\mathbf{e}_c)} =\frac{m_i}{B\tau_{\mathcal{E}}} \left[p_i^{\mathcal{E}}(c)-p_i^{\mathcal{M}}(c)\right].$$ Treating the scores as independent variables, a gradient step of size $\eta > 0$ on $\mathcal{L}_{\mathrm{LAMD}}$ changes the encoder margin from Eq. 10 by $$\begin{aligned} (\Delta_i^{\mathcal{E}})' - \Delta_i^{\mathcal{E}} =\frac{\eta m_i}{B\tau_{\mathcal{E}}}\Big[& p_i^{\mathcal{M}}(c_i^+)-p_i^{\mathcal{M}}(c_i^{\mathrm{hard}})\\\\ &-p_i^{\mathcal{E}}(c_i^+)+p_i^{\mathcal{E}}(c_i^{\mathrm{hard}})\Big]. \end{aligned}$$ The prime denotes the margin after this score update. For a selected query, $m_i=1$ and $\Delta_i^{\mathcal{M}} > 0$. If the encoder misranks or ties the pair, $\Delta_i^{\mathcal{E}}\leq0$. Since softmax preserves pair ordering, the matcher probability gap is positive and the encoder probability gap is non-positive. The update therefore increases $\Delta_i^{\mathcal{E}}$ in score space, even though the loss covers the whole candidate list. More generally, this margin increases only when the matcher's probability gap exceeds the encoder's. A matcher can rank the pair correctly but have a smaller probability gap than an encoder that already separates it well; the KL term would then reduce that encoder margin. Eq. 11 excludes queries with $\Delta_i^{\mathcal{E}}\geq\delta_{\mathcal{E}}$ from matching distillation. For correctly ranked pairs below the predefined threshold $\delta_{\mathcal{E}}$, the direction still depends on the probability-gap condition above. Nor does this pairwise check establish that all other matcher comparisons are correct. The analysis explains the local correction on the checked pair; Table B in our previous reply evaluates the complete selection rule. Throughout training, $\mathcal{L}_{\mathrm{con}}$ continues to provide label-based supervision for all queries.
Reviewer_xd4SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MATE trains a multimodal bi-encoder through latent reasoning distillation and matching distillation. It compresses a multi-step teacher into one latent step using an answer-guided summary of intermediate states, then transfers selected ranking preferences from a matcher over latent states and pooled embeddings. Both auxiliary models are removed at inference. On MMEB-V2, it reports overall scores of 65.4 and 69.7 at the 2B and 7B scales, with ablations on distillation targets, matcher inputs, and negative counts.

Critical strengths

1.The approach addresses a practical trade-off between retrieval quality and encoding cost. Reasoning and pairwise matching are used during training, while inference retains independent single-vector encoding and compatibility with a precomputed corpus index. 2.The ablations support the main components. Answer-guided distillation improves over uniform trajectory averaging by 1.8 points in Table 5. The matched-budget InfoNCE continuation in Table 4 also helps establish that the matching gains do not simply come from additional training.

Critical weaknesses

1.Important training details are unclear in the provided manuscript. Section 3.3 relies on explicit rationales and supervised answers, but their sources and construction across retrieval tasks are not specified. The training mixture also needs clarification, particularly when interpreting the modest gains over the strongest baselines. 2.The specific roles of progressive compression and selective matching are not isolated. Table 5 compares target representations, rather than progressive versus direct compression. Table 4 varies matcher inputs and negative counts, but does not compare the gate with applying the same distillation loss to every query.

Questions for authors

1.Which datasets and rationale sources train the teacher, and how are supervised answers defined for queries and candidates in non-QA retrieval tasks? 2.Are comparisons with direct one-step compression and ungated matching available? Please also clarify the compression schedule and threshold selection.

1 comment
AuthorsOpenReview ↗

Training Details and Ablation Studies

Thank you for recognizing MATE's practical benefits. On MMEB-V2, progressive compression gives the one-step student a 4.5-point gain over direct compression. In a separate comparison, the encoder scores 1.4 points higher with matching supervision on selected queries than on all queries. We have marked the changes in blue in the revised paper and supplement.

1. Training data and what “answer” means in non-QA tasks

We train the teacher on the public UME-R1 supervised fine-tuning corpus. It already provides rationales and answers for both queries and candidates. Our training mix covers 22 image or document datasets and three video datasets. We use subset-balanced sampling and reuse the same training pairs across stages. The revised supplement lists the datasets and gives annotation examples (Sections B and C).

The retrieval label tells us which candidate is correct. The <answer> field is a short text summary. It can name a category in classification or describe an object in a matching image. In document retrieval, it can summarize a page. The page itself is what the model must retrieve. A query annotation can use a paired answer as context when one is available. Candidates are annotated from their own content alone.

To form the student's target, we weight the teacher's reasoning states by their similarity to the answer representation. The retrieval embedding comes from states that precede the answer text. At inference, MATE uses only the one-step encoder to embed each query or candidate.

2. Progressive versus direct compression

The “One-step baseline” in Table 3 of the original submission is our direct-compression control. Both students start from the same checkpoint. We keep the teacher and answer-guided target fixed and use the same data, losses, and number of student updates. Direct compression uses one latent step throughout Stage 2. With progressive Latent Reasoning Distillation (LRD), we use eight steps for 20% of the updates, four for 20%, two for 25%, and one for the final 35% (Supplementary Table 1).

Table A. Direct and progressive compression on MMEB-V2 before matching distillation (Qwen2-VL-2B). Overall is the unweighted mean across all 78 tasks.

MethodInference stepsOverall
Teacher864.7
Direct compression159.5
Progressive LRD164.0

3. Does query selection help?

We have added a control that applies the matching loss to every query. All three runs in Table B start from the same LRD checkpoint. Each uses the contrastive retrieval loss (InfoNCE) and trains for the same number of updates.

Both distillation runs use the same frozen matcher, temperatures, and KL weight. The matcher scores one positive and four hard negatives per query. We average the KL loss over the full batch (Eq. 14). Unselected queries contribute only to InfoNCE.

Table B. Effect of query selection on MMEB-V2 (Qwen2-VL-2B).

Matching supervisionImageVideoVisual documentsOverall
None (InfoNCE only)68.245.171.563.9
All queries67.946.671.164.0
Selected queries69.747.372.665.4

Using the matcher on every query improves the video score but lowers the image and document scores compared with InfoNCE alone. The overall score changes little, from 63.9 to 64.0. Selecting queries improves all three averages over the all-query run and brings the overall score to 65.4.

4. Why these thresholds?

For each query, we compare the positive with the negative that the encoder scores highest among the four. We use matching supervision only when both conditions in Eq. 11 hold. The matcher must rank the positive strictly higher, so we set its threshold to zero (δ_M = 0). The encoder's margin (positive score minus negative score) must be below δ_E = 0.05. This includes errors, ties, and correct rankings with a small lead. A cutoff of zero would keep only errors. At the encoder's distillation temperature of 0.05, a cosine similarity gap of 0.05 corresponds to one logit unit (Eq. 13).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.