- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MATE, which consists of Latent Reasoning Distillation (LRD) and Latent-Aware Matching Distillation (LAMD). LRD trains a multi-step latent teacher with explicit rationales and progressively distills its trajectory into a single latent state, while LAMD uses a training-only matcher to examine latent states and pooled embeddings of queries and candidates and transfers listwise matching preferences to a bi-encoder.
Critical strengths
- LRD enables the teacher's multi-step reasoning traces to be distilled into a single latent state. In addition, LAMD uses a matcher only during training to align the latent state and embeddings of queries and candidates provide ranking supervision for the bi-encoder so that the candidates are ranked by embeddings of query and candidates at inference time.
- The authors compared MATE with recent UME models across discriminative, explicit-reasoning, and latent-reasoning approaches on the 78 tasks included in MMEB-V2, and demonstrates improved average performance.
Critical weaknesses
- The paper claims that the 2B MATE model achieves an overall score of 65.4 on the 78-task MMEB-V2 benchmark while delivering 62× higher throughput than UME-R1 under matched inference settings. Thus, a key contribution is not only improved effectiveness but also efficiency. However, the inference setup for the throughput comparison is not clearly described, making it difficult to assess the fairness and reproducibility. Additionally, the paper lacks further analysis of the throughput results beyond Figure 1.
- There are several experimental results that are not sufficiently explained or discussed.
Questions for authors
-
The paper does not analyze the performance impact of discarding the matcher during inference. Reporting results with the matcher used at inference time would provide a useful upper bound and help quantify the trade-off between effectiveness and efficiency.
-
In Figure 1, the x-axis label appears to be misspelled as "Throughout". It should likely be "Throughput".
-
The MMEB-V2 results reported in Figure 1 do not appear to match the Avg scores in Table 1. It would be helpful to clarify how these results were obtained and why the values differ, so that readers can clearly understand whether the discrepancy arises from different evaluation settings.
-
Figure 2 is not discussed in the main text. In addition, to the best of my knowledge, the categorization and illustrations in Figure 2(a)-(c) are almost identical to those presented in LaME. Since these concepts do not appear to have been first introduced in this work, it would be helpful to explicitly acknowledge the prior work and clarify the relationship between this paper and LaME.
-
The largest gains in Table 1 are observed on GD in Image and MERT in video. Further analysis would help clarify when the proposed method is most effective. Table 2 includes a comparison between IND and OOD settings intentionally, but the results are not discussed in the paper.
Inference Efficiency and Result Analysis
Thank you for recognizing MATE's effectiveness and practical benefits. Our single-step encoder retains the gains from reasoning and matching supervision during training. The new comparison below shows that retaining the matcher adds only 0.1 points while increasing latency by 37%.
1. Inference setup and throughput
We measure query-encoding throughput with batch size 1, BF16, FlashAttention-2, and no quantization. MATE and UME-R1 use the same hardware and inference settings. MATE achieves 6.24 ± 0.02 samples/s, compared with 0.10 ± 0.01 samples/s for UME-R1, corresponding to approximately 62× the throughput.
Timing includes image loading, preprocessing, and query embedding generation, with CUDA synchronization. It excludes model loading, warm-up, offline corpus encoding, search, and reranking. MATE uses one latent step and avoids generating a textual rationale before producing an embedding. Neither its teacher nor its matcher is needed at inference. Further implementation details are provided in Supplementary Section B.
2. What happens if the matcher is retained?
We compare the final Qwen2-VL-2B encoder with and without reranking. Both modes rank the original candidate pool by embedding similarity. The second then uses the matcher to rerank the top 16 candidates. The encoder was trained with four hard negatives. The encoder and matcher remain fixed, and the training selection rule is not used at inference.
Table A. Effect of retaining the matcher on MMEB-V2.
Both modes use the same queries, candidate pools, hardware, precision, and query batch size. Candidate representations are cached offline, and each query is encoded once. Latency runs from prepared inputs to the final ranking, including query encoding, candidate scoring, and reranking when enabled. Its scope differs from the throughput measurement above.
Reranking gains 0.1 points and adds 68 ms/query. In this top-16 setting, the default encoder achieves nearly the same score at lower latency. The paper's matching gain, from 64.0 to 65.4, already uses no matcher at inference.
3. Figure 1 and the relationship to LaME in Figure 2
Figure 1 was assembled manually, so some markers were not positioned precisely at their corresponding values. The underlying results are the 78-task overall scores in the All column of Table 1. We will correct the marker positions to match these values. The x-axis label has also been corrected to “Throughput”.
Figure 2 was independently designed to compare established retrieval approaches with MATE. Panels (a)–(c) summarize existing paradigms, while panel (d) illustrates our method. We discuss the figure in the main text and clarify the methodological differences between MATE and related work, including LaME. LaME trains latent tokens with reconstruction and retrieval objectives. MATE trains an eight-step latent teacher, compresses it to one step through LRD, and uses LAMD to transfer matching supervision into the encoder. Panel (d) will make this contribution clearer.
4. Grounding, moment retrieval, and IND/OOD results
At the 2B scale, MATE improves over LaME by 10.3 points on grounding and 7.4 points on moment retrieval . These tasks require identifying a particular object or temporal segment among candidates that share much of their visual content. Retaining relevant attributes, relations, and actions is therefore useful, as the examples in Supplementary §E illustrate.
The ablations support both training stages: answer-guided supervision raises the video score from 41.4 to 44.9 over final-state supervision, and matching distillation further raises it to 47.3. This is consistent with preserving useful content during compression and then improving discrimination between similar candidates.
On the image OOD split, MATE scores 67.2 vs. 64.4 for LaME; on IND, it scores 71.6 vs. 73.1. We will discuss both results explicitly. The gain over LaME on this image benchmark comes from the OOD tasks, while LaME remains stronger on the IND tasks.
We will add these details to clarify when MATE helps and how its inference cost is measured.