acceptodds
Under review as a conference paper at ICLR 2027

Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings

10 views

Abstract

Universal multimodal embedding (UME) maps multimodal inputs into a shared embedding space for diverse retrieval tasks. Recent methods improve embeddings through Chain-of-Thought (CoT) reasoning optimized with GRPO using retrieval rewards. However, existing methods overlook the mismatch between candidate-aware retrieval supervision and input-only CoT generation: (1)trajectory-level rewards convey retrieval outcomes without explicitly identifying the input-supported evidence that distinguishes the positive from hard negatives; (2) input-only generation cannot directly assess whether further reasoning improves retrieval, potentially producing redundant CoTs with substantial latency. To bridge this gap, we propose Reason What Matters (ReWAM), a retrieval-grounded framework that aligns candidate-aware supervision with input-only generation. Specifically, we introduce Retrieval-Aware Self-Distillation (RASD), which extracts privileged guidance from input-supported facts and evidence distinguishing the positive from hard negatives. Conditioned on this guidance, an on-policy self-teacher provides token-level feedback to refine credit assignment, directing policy updates toward retrieval-relevant reasoning grounded in the input. We further propose Retrieval-Adaptive Inference (RAI), which learns a retrievalaware stopping criterion from prefix-level retrieval feedback. It stops redundant reasoning without candidate access and uses speculative decoding to further reduce CoT latency. Extensive experiments on MMEB-V2 and MRMR demonstrate that ReWAM achieves state-of-the-art retrieval performance while delivering up to 5× the inference throughput of competitive explicit-CoT UME methods. ReWAM thus enables high-quality retrieval through efficient input-only reasoning, making explicit CoT practical for corpus-scale multimodal retrieval. The code will be publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal multimodal embedding, reasoning, retrieval-grounded

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BjeyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ReWAM addresses the mismatch between candidate-aware training rewards and input-only reasoning in a decoupled reasoner--embedder system. Retrieval-Aware Self-Distillation (RASD) uses input evidence, positive--negative distinctions and claim verification to reweight GRPO advantages at the token level. Retrieval-Adaptive Inference (RAI) predicts remaining retrieval utility and combines early stopping with speculative decoding. The strongest configurations reach 67.4/69.0 on MMEB-V2 and 53.3 on MRMR.

Critical strengths
  1. The motivation is clear, and some observations are insightful. A successful retrieval trajectory can still contain unsupported or irrelevant claims, so using grounded candidate comparisons to refine its training signal is a sensible idea.

  2. The evaluation is broad and comprehensive, covering 78 MMEB-V2 datasets and 11 MRMR subtasks across several embedding backbones.

Critical weaknesses
  1. The highest accuracy and the main acceleration results come from different configurations, leaving the final system's joint accuracy and efficiency unverified. Appendix B.4 explicitly states that RAI is not enabled in the main retrieval tables. The detailed acceleration experiments use only the Qwen2-VL-2B embedder and do not cover all configurations achieving the main results or the MRMR benchmark. The authors should clarify this mismatch. The overall pipeline is also quite complex, and I would like to better understand its efficiency and whether its performance gains are attributable to external modules or additional parameters.

  2. The authors identify two issues in current reasoning-based embedding methods: the mismatch between candidate-aware supervision and input-only reasoning, and the potential redundancy in reasoning. These observations are insightful, but existing methods, such as MMEmb-R1[1], TRACE[2], and Think When Needed[3], also address similar issues. In particular, the two problems targeted by MMEmb-R1 are largely the same as those considered in this paper. The authors should compare against these methods, or at least discuss their similarities and differences, to better establish the paper's contribution.

[1] Yuchi Wang et al. MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control. 2026.

[2] Xiangzhao Hao et al. TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval. 2026.

[3] Longxiang Zhang et al. Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture. 2026.

Questions for authors

The experiments do not directly demonstrate that RASD makes reasoning more grounded or reduces unsupported content. The current evidence mainly consists of overall retrieval scores, without a fine-grained analysis of whether RASD actually reduces unsupported content.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Bjey (1/2): W1

We thank you for your feedback on the joint evaluation, related methods, and grounding evidence. Our responses include paired accuracy–throughput results, an expanded comparison with prior work, and direct CoT visualizations. We hope these additions resolve your concerns and would sincerely appreciate a more favorable score.

Weaknesses

W1. Joint accuracy-efficiency evaluation and pipeline cost

The highest accuracy and the main acceleration results come from different configurations, leaving the final system's joint accuracy and efficiency unverified. Appendix B.4 explicitly states that RAI is not enabled in the main retrieval tables. The detailed acceleration experiments use only the Qwen2-VL-2B embedder and do not cover all configurations achieving the main results or the MRMR benchmark. The authors should clarify this mismatch. The overall pipeline is also quite complex, and I would like to better understand its efficiency and whether its performance gains are attributable to external modules or additional parameters.

1、Response.

We thank the reviewer for raising this concern. Our results in Tables 1–2 evaluate retrieval quality using full CoTs without RAI, whereas Table 6 isolates RAI's inference-time effects with the Qwen2-VL-2B embedder (Appendix B.4). To directly assess accuracy and speed under the same model configurations, we have added comparisons with RAI disabled and enabled in Tables 12–13. These additional experiments cover three MMEB-V2 embedders and our strongest MRMR model. The draft length is fixed at K=7. Each pair below shows without RAI → with RAI, and speedups are computed from the reported throughput values.

MMEB-V2: paired retrieval accuracy and throughput (Appendix C.1, Table 12).

EmbedderModalityScoreSamples/sSpeedup
Qwen3-VL-2BImage71.9 → 71.60.9 → 3.33.67×
Qwen3-VL-2BVisDoc74.2 → 73.80.8 → 3.24.00×
Qwen3-VL-2BVideo49.7 → 49.30.8 → 3.44.25×
Qwen3-VL-4BImage73.4 → 73.10.8 → 2.93.63×
Qwen3-VL-4BVisDoc75.8 → 75.40.7 → 2.13.00×
Qwen3-VL-4BVideo49.8 → 49.60.8 → 2.12.63×
Qwen2-VL-7BImage74.2 → 73.90.8 → 2.73.38×
Qwen2-VL-7BVisDoc75.7 → 75.50.7 → 1.92.71×
Qwen2-VL-7BVideo49.5 → 49.20.7 → 2.13.00×

MRMR: Qwen3-VL-4B (Appendix C.1, Table 13).

DomainScoreSamples/sSpeedup
Knowledge71.9 → 70.91.6 → 2.81.75×
Theorem47.9 → 47.02.4 → 4.11.71×
Contradiction35.6 → 35.21.6 → 2.91.81×
Overall53.3 → 52.41.8 → 3.11.72×

Across the three MMEB-V2 embedders, RAI provides 2.6–4.3× throughput with 0.2–0.4-point score decreases. RAI is trained on general-purpose MMEB data. MRMR instead focuses on reasoning-intensive retrieval evaluation, introducing a shift in the query distribution. Under this shift, RAI still achieves 1.72× throughput with only a 0.9-point score decrease. The accelerated score of 52.4 still exceeds RIME’s 50.2 (Table 2), retaining state-of-the-art retrieval performance. This demonstrates a favorable accuracy–speed trade-off, supporting its generalization to reasoning-intensive retrieval.

We clarify that RASD improves the existing reasoner through token-level training supervision, without enlarging the reasoner or embedder backbone (Tables 3–5). We evaluate inference acceleration over the complete encoding pipeline, including CoT generation, embedding extraction, and the overhead of RAI's utility head and draft model (Appendix B.4). The observed speedup therefore reflects the overall inference efficiency of the ReWAM paradigm.

AuthorsOpenReview ↗

Response to Reviewer Bjey (2/2): W2 and Q1


W2. Comparison with related adaptive reasoning methods

The authors identify two issues in current reasoning-based embedding methods: the mismatch between candidate-aware supervision and input-only reasoning, and the potential redundancy in reasoning. These observations are insightful, but existing methods, such as MMEmb-R1[1], TRACE[2], and Think When Needed[3], also address similar issues. In particular, the two problems targeted by MMEmb-R1 are largely the same as those considered in this paper. The authors should compare against these methods, or at least discuss their similarities and differences, to better establish the paper's contribution.

[1] Yuchi Wang et al. MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control. 2026.

[2] Xiangzhao Hao et al. TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval. 2026.

[3] Longxiang Zhang et al. Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture. 2026.

2、Response.

We thank the reviewer for recognizing the motivation behind our work and for pointing us to these closely related studies. We agree that they share the broad goal of making reasoning useful and efficient for multimodal retrieval.

MMEmb-R1 uses counterfactual pair-aware selection to identify beneficial reasoning paths and learns whether to invoke reasoning for an input. TRACE integrates structured CoT generation with embedding learning and exhibits adaptive behavior that bypasses reasoning for simpler queries. Think When Needed uses a shared-backbone dual-LoRA architecture and a routing gate to decide whether to generate CoT. These methods primarily address whether reasoning should be used, which trajectory to select, or how reasoning should be integrated with embedding learning.

However, these methods depend on sampled candidates when their adaptive decisions are trained on within-batch retrieval gains. This creates a risk of batch-specific shortcuts and weak generalization. We mitigate this risk through RASD, which uses input-grounded contrastive evidence to provide token-level supervision. It conditions an on-policy self-teacher on input-supported evidence and positive–negative decision boundaries to refine coarse trajectory-level retrieval feedback. This guides training toward supported, discriminative reasoning instead of uniformly reinforcing an entire successful CoT. RAI further adapts reasoning after generation begins, predicting remaining retrieval utility from each prefix to stop unnecessary continuation. It also accelerates token generation with speculative decoding. These designs extend adaptive reasoning beyond input-level routing and trajectory selection.

We have added these references to §2 and expanded the discussion of their shared motivations, limitations, and technical differences in Appendix E.

Questions

Q1. Grounding and unsupported content

The experiments do not directly demonstrate that RASD makes reasoning more grounded or reduces unsupported content. The current evidence mainly consists of overall retrieval scores, without a fine-grained analysis of whether RASD actually reduces unsupported content.

3、Response.

We thank the reviewer for raising this concern. We have added claim-level CoT comparisons in Appendix C.3 (Figs. 15–17) to make the grounding differences directly inspectable. Each comparison uses the same images and user prompt and shows complete greedy CoTs without RAI truncation.

CaseGRPO reasoningRASD reasoning
Fig. 15, MRMR MathIntroduces a square absent from the input and uses its supposed side length as the cone's base radius.Uses the sector's arc as the cone's base circumference and its radius as the slant height.
Fig. 16, MRMR PhysicsTreats 100% transformer efficiency as equal primary and secondary voltages, despite the given turns ratio.Applies the 1:20 turns ratio and power conservation specified by the input.
Fig. 17, MRMR EngineeringInterprets a simply supported beam as fixed at both ends and builds further calculations on this incorrect premise.Respects the pin–roller supports and triangular load shown in the input.

These examples show RASD-trained reasoning following input-supported constraints and avoiding the unsupported premises seen in the GRPO traces. The complete RASD CoTs are also shorter in these three cases, with 143 versus 431, 107 versus 388, and 135 versus 454 words, respectively. The visualizations therefore provide claim-level qualitative evidence of better grounding and less redundant reasoning, complementing the retrieval ablations in Tables 2 and 4.

Reviewer_P4YiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ReWAM, a framework for retrieval-grounded explicit reasoning in universal multimodal embeddings. Retrieval-Aware Self-Distillation (RASD) uses an analyzer to extract input-supported facts and distinctions between positives and hard negatives, then has an on-policy self-teacher produce token-level feedback to modulate trajectory-level retrieval advantages. Retrieval-Adaptive Inference (RAI) trains a utility head on prefix-level retrieval gains to stop reasoning when little additional utility is predicted, and adds speculative decoding. On MMEB-V2, the paper reports overall scores of 67.4 and 69.0 for its 2B and larger-model groups and up to 5x throughput over an explicit-CoT baseline. It also reports strong results on MRMR.

Critical strengths
  • RASD uses candidate-aware evidence during training to supervise reasoning grounded in the query, addressing the mismatch between candidate-comparison rewards and query-only inference. The design is well motivated, and experiments show retrieval-accuracy improvements.
  • RAI predicts the retrieval utility of reasoning prefixes and stops generation when the expected additional utility is low. This is a plausible way to reduce unnecessary reasoning, and experiments demonstrate inference-time efficiency gains.
Critical weaknesses
  • The main claims are potentially misleading because the paper does not demonstrate its retrieval-accuracy and throughput gains in the same configuration. More fundamentally, although RAI is applied to a RASD-trained reasoner, the paper does not show that RASD and RAI work synergistically beyond this sequential composition. Once the contribution of existing speculative decoding is separated, the paper's own gains in retrieval accuracy and inference speed appear limited relative to the strength of the claims.
    1. The retrieval-accuracy and throughput results use different configurations. Table 1 reports retrieval results without RAI (Appendix B.4). The 2B ReWAM scores (Image/VisDoc/Video: 71.6/72.8/47.7, with an overall score of 66.5) exactly match the no-RAI Baseline in Table 6. Conversely, the main throughput claim is based on inference-time acceleration. Thus, the paper does not establish that the reported SOTA retrieval accuracy and up-to-5× throughput are achieved together.
    2. The experiments do not establish a synergistic benefit from RASD and RAI. The main retrieval results evaluate RASD without RAI, while the RAI ablation tests inference-time changes only with the RASD-trained reasoner. Table 3 shows that RASD improves the aggregate from 65.7 to 66.5 (+0.8), but the paper does not compare RAI with and without RASD to isolate whether the two components interact synergistically.
    3. The gains attributable to the proposed components are modest, with a retrieval-accuracy/efficiency trade-off. Speculative decoding, an existing technique, alone gives 2.0–2.4× the autoregressive baseline throughput with nearly unchanged retrieval accuracy. The proposed Retrieval Utility Head adds 26–64% throughput on top of that but lowers retrieval accuracy by 0.2–0.4 points. Full RAI at K=7 scores 66.1 overall versus 66.5 without RAI (Tables 6 and 9). On MMEB-V2, several closest model-size comparisons are also narrow (69.0 vs. 68.6 at 7B, and 68.7 vs. 68.1 at 4B), although some other comparisons and MRMR results show larger gains. These results do not support the abstract and introduction's broad impression of a substantial, unified retrieval-accuracy and efficiency advantage. The authors should report matched results for RASD+RAI, separate the gains from speculative decoding and the proposed stopping head, and qualify the main claims.
  • RAI is one of the paper's central proposed components. Tables 6 and 9 show that it increases throughput with only a modest drop in retrieval accuracy in the tested MMEB-V2 setup, providing evidence of efficacy in that setting. However, these results do not sufficiently characterize its sensitivity, robustness, or generalization. The utility head is trained on prefixes from a single greedy rollout per input, and inference uses a fixed stopping threshold (τ_stop = 0.01). The paper should further analyze:
    1. Threshold sensitivity and calibration. The paper does not report a systematic sweep of τ_stop. Accuracy-throughput curves, utility calibration, and stopping-length distributions across thresholds would show how dependent the results are on this choice and how accuracy and throughput trade off for a given trained utility head.
    2. Prefix-wise stability and first-crossing reliability. The paper does not show how utility predictions evolve across consecutive prefixes: whether they are smooth or monotonic, whether predictions stay below threshold after the first crossing, or whether useful reasoning tokens and higher utility can appear later. Since decoding stops at the first below-threshold prediction, the authors should report how often this decision is followed by a utility rebound or a retrieval-improving continuation.
    3. Generalization and robustness. Because training uses one greedy rollout per input, evaluation on held-out tasks or benchmarks, shifted query distributions, and alternative or stochastic decoding paths is needed to establish whether the reported trade-off transfers beyond the tested setup. These analyses would clarify when RAI's stopping policy is reliable and when it may prematurely truncate useful reasoning.
Questions for authors

Please refer to the Weaknesses section for the specific questions and requests for additional analyses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer P4Yi (1/4): W1–W2

Thank you for your detailed comments on the accuracy–efficiency trade-off and experimental design. We have clarified the settings and component contributions and added robustness analyses. Should this evidence address the issues you raised, we would greatly appreciate your reconsideration of the score.

Weaknesses-part1

Overall concern.

The main claims are potentially misleading because the paper does not demonstrate its retrieval-accuracy and throughput gains in the same configuration. More fundamentally, although RAI is applied to a RASD-trained reasoner, the paper does not show that RASD and RAI work synergistically beyond this sequential composition. Once the contribution of existing speculative decoding is separated, the paper's own gains in retrieval accuracy and inference speed appear limited relative to the strength of the claims.

Response overview.

We thank the reviewer for raising these concerns. RASD and RAI serve complementary roles in improving retrieval-relevant reasoning and reducing unnecessary inference cost. Below, we clarify the evaluation settings and provide paired accuracy–throughput results and component ablations.


W1. Matched retrieval accuracy and throughput

The retrieval-accuracy and throughput results use different configurations. Table 1 reports retrieval results without RAI (Appendix B.4). The 2B ReWAM scores (Image/VisDoc/Video: 71.6/72.8/47.7, with an overall score of 66.5) exactly match the no-RAI Baseline in Table 6. Conversely, the main throughput claim is based on inference-time acceleration. Thus, the paper does not establish that the reported SOTA retrieval accuracy and up-to-5× throughput are achieved together.

1、Response.

As specified in Appendix B.4, Tables 1 and 2 report full-CoT retrieval performance without RAI. The “Baseline” in Table 6 is therefore the same RASD-trained ReWAM configuration with the Qwen2-VL-2B embedder reported in Table 1. Table 11 already provides paired accuracy–throughput results for this model with and without RAI:

Inference settingMMEB-V2 overallThroughput (samples/s)
Without RAI66.51.1
With RAI (K=7)66.13.7

Enabling RAI yields approximately 3.4× the throughput with a 0.4-point score decrease, without retraining the reasoner or embedder. The accelerated score remains 2.0 points above RIME (66.1 vs. 64.1) with the same Qwen2-VL-2B embedder backbone.

The 5× and 3.4× figures use different comparison baselines. The abstract explicitly states that the up-to-5× throughput gain is relative to “competitive explicit-CoT UME methods.” Fig. 1(e) provides this cross-method comparison: ReWAM achieves 3.75 samples/s versus 0.75 for Embed-RL under the same single-H800, batch-size-1 evaluation protocol. The approximately 3.4× gain in Table 11 compares ReWAM with and without RAI. The SOTA retrieval results refer to the full-CoT configurations in Tables 1 and 2; the paired results above quantify the accelerated configuration's accuracy–speed trade-off.


W2. Interaction between RASD and RAI

The experiments do not establish a synergistic benefit from RASD and RAI. The main retrieval results evaluate RASD without RAI, while the RAI ablation tests inference-time changes only with the RASD-trained reasoner. Table 3 shows that RASD improves the aggregate from 65.7 to 66.5 (+0.8), but the paper does not compare RAI with and without RASD to isolate whether the two components interact synergistically.

2、Response.

We thank the reviewer for raising this question. RASD is a training method for the reasoner, while RAI is an inference-time procedure applied to the trained reasoner. RASD uses candidate-aware evidence to provide token-level supervision and improve retrieval-relevant reasoning. After RASD training, the reasoner remains frozen, and RAI uses predicted retrieval gains to stop unnecessary CoT continuation.

We clarify that RASD and RAI are not designed as a synergistic pair. They serve separate roles during training and inference. Comparing RAI with and without RASD is therefore unnecessary for validating their respective contributions.

The ablations follow this separation between training and inference. Table 3 measures the benefit of RASD training with RAI disabled and shows an overall score increase from 65.7 to 66.5. Table 6 evaluates inference-time acceleration on the fixed RASD-trained reasoner by comparing variants with and without the Retrieval Utility Head at each draft length. Table 6/11 reports the accuracy–throughput trade-off of applying RAI to the same trained model. These experiments therefore demonstrate RASD's training benefit and RAI's additional inference benefit for the trained ReWAM model. We have clarified the settings of Tables 3, 6, and 11 in Appendix B.4 to distinguish training ablations from inference-time evaluations.

AuthorsOpenReview ↗

Response to Reviewer P4Yi (2/4): W3


W3. Contributions beyond speculative decoding

The gains attributable to the proposed components are modest, with a retrieval-accuracy/efficiency trade-off. Speculative decoding, an existing technique, alone gives 2.0–2.4× the autoregressive baseline throughput with nearly unchanged retrieval accuracy. The proposed Retrieval Utility Head adds 26–64% throughput on top of that but lowers retrieval accuracy by 0.2–0.4 points. Full RAI at K=7 scores 66.1 overall versus 66.5 without RAI (Tables 6 and 9). On MMEB-V2, several closest model-size comparisons are also narrow (69.0 vs. 68.6 at 7B, and 68.7 vs. 68.1 at 4B), although some other comparisons and MRMR results show larger gains. These results do not support the abstract and introduction's broad impression of a substantial, unified retrieval-accuracy and efficiency advantage. The authors should report matched results for RASD+RAI, separate the gains from speculative decoding and the proposed stopping head, and qualify the main claims.

3、Response.

We thank the reviewer for this assessment. We agree that speculative decoding is an established technique. We do not propose a new speculative-decoding algorithm. Our contribution is an acceleration system for reasoning-enhanced UME that combines speculative decoding with retrieval-aware stopping. RAI improves inference efficiency along two complementary dimensions: speculative decoding reduces the cost of generating each CoT token, while the Retrieval Utility Head avoids redundant CoT continuation.

The proposed stopping mechanism contributes substantial acceleration beyond speculative decoding alone. Baseline uses the RASD-trained reasoner with autoregressive decoding and without RAI. The accelerated variants use speculative decoding alone or with the Retrieval Utility Head. All variants use the same reasoner and Qwen2-VL-2B embedder. Both accelerated variants use K=7. Throughput and absolute gains are in samples/s (Table 6):

ModalityBaselineSpeculative decodingGainWith utility headExtra gainGain ratio
Image1.22.9+1.74.5+1.694.1%
VisDoc1.12.3+1.23.3+1.083.3%
Video1.12.5+1.43.4+0.964.3%

Speculative decoding adds 1.2–1.7 samples/s, and the head adds a further 0.9–1.6 samples/s. The head's added gain is therefore 64.3–94.1% of the gain from speculative decoding alone. This controlled comparison shows a substantial contribution beyond the existing acceleration technique.

We acknowledge the accuracy–speed trade-off, but the accelerated model remains strong. At K=7, RAI raises overall throughput from 1.1 to 3.7 samples/s (approximately 3.4×), while the MMEB-V2 score changes from 66.5 to 66.1, a 0.4-point decrease (Table 11). Even after acceleration, it exceeds RIME (64.1), the strongest reported Qwen2-VL-2B baseline in Table 1, by 2.0 points. This shows a favorable accuracy–throughput trade-off for the accelerated model. Moreover, the full-CoT retrieval advantage is more pronounced on reasoning-intensive MRMR, where ReWAM-4B exceeds Embed-RL-4B by 6.5 points (53.3 versus 46.8, Table 2).

AuthorsOpenReview ↗

Response to Reviewer P4Yi (3/4): W4–W5

Weaknesses-part2

Overall concern.

RAI is one of the paper's central proposed components. Tables 6 and 9 show that it increases throughput with only a modest drop in retrieval accuracy in the tested MMEB-V2 setup, providing evidence of efficacy in that setting. However, these results do not sufficiently characterize its sensitivity, robustness, or generalization. The utility head is trained on prefixes from a single greedy rollout per input, and inference uses a fixed stopping threshold (τ_stop = 0.01). The paper should further analyze:

Response overview.

We thank the reviewer for identifying these important gaps. Tables 6 and 11 demonstrate RAI’s accuracy–throughput trade-off at τstop=0.01, but do not fully characterize when its stopping decisions remain reliable. We have therefore added analyses of threshold sensitivity, first-crossing behavior, and performance beyond the original evaluation setting.


W4. Stopping-threshold sensitivity and calibration

Threshold sensitivity and calibration. The paper does not report a systematic sweep of τ_stop. Accuracy-throughput curves, utility calibration, and stopping-length distributions across thresholds would show how dependent the results are on this choice and how accuracy and throughput trade off for a given trained utility head.

4、Response.

Threshold sensitivity. RAI stops when predicted remaining retrieval utility falls below τstop; a larger threshold therefore triggers earlier stopping. On MMEB-V2 Image, we obtain:

tauHit@1tokenssamples/s
0.00571.488.23.6
0.0171.387.94.5
0.0569.553.55.4
0.169.044.26.3

The selected threshold, 0.01, balances retrieval accuracy and speed. More aggressive thresholds shorten the CoT further, but incur a substantially larger accuracy cost. We have added these threshold-sensitivity results to Table 7.


W5. Prefix stability and first-crossing reliability

Prefix-wise stability and first-crossing reliability. The paper does not show how utility predictions evolve across consecutive prefixes: whether they are smooth or monotonic, whether predictions stay below threshold after the first crossing, or whether useful reasoning tokens and higher utility can appear later. Since decoding stops at the first below-threshold prediction, the authors should report how often this decision is followed by a utility rebound or a retrieval-improving continuation.

5、Response.

We thank the reviewer for raising this concern. Our utility predictions are not constrained to be monotonic. To examine first-crossing behavior, we generated full-CoT diagnostic rollouts with stopping disabled and evaluated the utility head along each trajectory using the paper’s threshold, τstop=0.01. We report the fraction of trajectories that would stop before natural completion and, among those with a continuation, the fraction whose predicted utility subsequently exceeds the threshold:

Rate (%)ImageVisDocVideoAll
Early stopping99.95481.26299.98696.311
Prediction rebound99.93697.27499.99899.521

Useful retrieval evidence is not necessarily concentrated in the initial CoT prefix. It may emerge at different positions as reasoning unfolds. Therefore, such threshold rebounds are an expected outcome. RAI aims to remove redundant continuation while preserving retrieval quality, but some useful later reasoning may still be truncated. This is consistent with the slight retrieval-score decrease after acceleration, reflecting the trade-off between retaining potentially useful reasoning and reducing generation cost.

Compared with fixed-length truncation, RAI better preserves retrieval quality at similar throughput (Table 11). At K=4, RAI scores 66.2 versus 64.2 for a fixed 64-token budget, with throughput of 3.4 versus 3.7 samples/s. This 2.0-point advantage demonstrates the effectiveness of using remaining retrieval utility to guide RAI's adaptive stopping.

AuthorsOpenReview ↗

Response to Reviewer P4Yi (4/4): W6


W6. Generalization and decoding robustness

Generalization and robustness. Because training uses one greedy rollout per input, evaluation on held-out tasks or benchmarks, shifted query distributions, and alternative or stochastic decoding paths is needed to establish whether the reported trade-off transfers beyond the tested setup. These analyses would clarify when RAI's stopping policy is reliable and when it may prematurely truncate useful reasoning.

6、Response.

We thank the reviewer for this suggestion. RAI is trained on CoT rollouts sampled from general-purpose multimodal retrieval training inputs (§4.1), whereas MRMR focuses on reasoning-intensive retrieval. The two settings therefore emphasize different query distributions. We evaluate cross-benchmark transfer on MRMR and decoding robustness on MMEB-V2 Image.

We use the Qwen3-VL-4B embedder. Detailed domain results are reported in Appendix C.1 Table 13. Each pair shows without RAI → with RAI:

DomainScoreThroughput (samples/s)Speedup
Knowledge71.9 → 70.91.6 → 2.81.75×
Theorem47.9 → 47.02.4 → 4.11.71×
Contradiction35.6 → 35.21.6 → 2.91.81×
Overall53.3 → 52.41.8 → 3.11.72×

Speedup is the throughput with RAI divided by the throughput without RAI, computed from the reported values. On MRMR, RAI increases throughput by 72.2%, from 1.8 to 3.1 samples/s, while the score decreases by 0.9 points. Even with acceleration, ReWAM-4B still outperforms the baseline methods in Table 2, exceeding Embed-RL-4B by 5.6 points (52.4 vs. 46.8) and RIME by 2.2 points (52.4 vs. 50.2). These results show that RAI transfers to a different query distribution while maintaining a favorable retrieval-accuracy–throughput trade-off.

Decoding robustness (MMEB-V2 Image).

TemperatureHit@1 (%)Throughput (samples/s)
071.34.5
0.370.123.99
0.670.093.95
1.070.173.87

The accelerated configuration remains robust across the tested decoding temperatures: compared with greedy decoding, Hit@1 falls by only 1.58%–1.70% at temperatures 0.3–1.0, while both Hit@1 (70.09–70.17) and throughput (3.87–3.99 samples/s) vary little across those nonzero temperatures. Together, the MRMR transfer and temperature results support RAI’s generalization and decoding robustness within these tested settings. The MRMR results are included in Appendix C.1 (Table 13), and the decoding-temperature analysis in §4.3 (Table 8).

Reviewer_gdfgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ReWAM, a think-then-embed-style UME model, with a standalone reasoner and an embedder trained separately. It further leverages a larger MLLM (e.g. Qwen3.5-122B-A10B) to provide guidance on whether the CoT is discriminative towards hard negatives. Lastly, to improve inference speed, it proposes to train an early stopping classifier to stop when the existing CoT is discriminative enough, and proposes to use speculative decoding to accelerate inference.

Critical strengths
  1. The paper achieves SOTA performance on both MMEB V2 and MRMR, under the models trained with MMEB V2 data.
Critical weaknesses
  1. Unfair comparison in Table 1. It seems the paper uses a 8B model as a reasoner, while still comparing with models of 2B sizes.
  2. Marginal improvement over Embed-RL. As shown in Table 1, ReWAM's improvement over its direct baseline, Embed-RL, is quite small (within 1%).
  3. Some major contributions have marginal effects. For instance, the proposed RASD has maximumly 0.5% improvement over GRPO (Table 4), the proposed CEPI also has 0.5% improvement over baseline (table 5).
  4. The inference speed comparison is not totally fair. First, this paper uses vLLM to accelerate generation, while previous works like Embed-RL did not. Second, it uses speculative decoding, which is a standard technique in inference acceleration and cannot be seen as a contribution here. Third, it uses a batch size of 1 for inference, which is advantageous to its framework with separate reasoner and embedder (compared to e.g. PLUME, UME-R1).
Questions for authors
  1. Regarding the RAI, could we not integrate the "early stopping" into the RL stage? E.g. to reward shorter sequences. Training an extra early stopping gate requires one extra stage, and will always impair performance, as the frozen embedder is trained with full sequences.
  2. Could you also show the throughput/QPS results and compare with e.g. UME-R1, PLUME, TTE-Flash etc.?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gdfg (1/4): W1–W2

We appreciate your careful review and constructive suggestions. Below, we clarify the model-size comparisons and evaluation protocol and provide additional ablation results. If these responses resolve your concerns, we would be grateful for your consideration of a higher score.

Weaknesses

W1. Fairness of model-size comparisons

Unfair comparison in Table 1. It seems the paper uses a 8B model as a reasoner, while still comparing with models of 2B sizes.

1、Response.

We thank the reviewer for pointing out the ambiguity in our model-size labels. In Table 1, the size refers to the embedder backbone. ReWAM additionally uses Qwen3-VL-8B-Instruct as reasoner, as stated in §4.1. We have clarified this distinction in the table caption.

We use Qwen3-VL-8B-Instruct (§4.1) to match the reasoner backbone of Embed-RL (ECCV 2026), our closest decoupled baseline. We also compare the same Qwen3-VL-2B and Qwen3-VL-4B embedder backbones. The comparisons below therefore match the model families and backbone sizes of both components:

MethodReasoner backboneEmbedder backboneMMEB-V2 overall
Embed-RLQwen3-VL-8B-InstructQwen3-VL-2B66.8
ReWAMQwen3-VL-8B-InstructQwen3-VL-2B67.4
Embed-RLQwen3-VL-8B-InstructQwen3-VL-4B68.1
ReWAMQwen3-VL-8B-InstructQwen3-VL-4B68.7

Embed-RL also compares its decoupled framework with non-decoupled models on MMEB-V2. We match its reasoner and embedder backbone configuration and improve the overall score by 0.6 points at both embedder scales. Results from non-decoupled frameworks in Table 1 provide a broader performance reference.


W2. Improvements over Embed-RL

Marginal improvement over Embed-RL. As shown in Table 1, ReWAM's improvement over its direct baseline, Embed-RL, is quite small (within 1%).

2、Response.

We thank the reviewer for this observation. MMEB-V2 is a challenging benchmark of general-purpose multimodal retrieval across 78 datasets. ReWAM improves over the strong Embed-RL baseline by 0.6 points at both the 2B and 4B embedder scales under matched backbones (Table 1). This consistent gain at both scales is a meaningful improvement in general retrieval capability. However, the MMEB-v2 overall average does not specifically isolate improvements in retrieval reasoning.

RASD uses input-grounded contrastive evidence to refine token-level credit assignment. This reduces reinforcement of unsupported claims and emphasizes evidence that distinguishes relevant targets from hard negatives. The retrieval gains are more pronounced on MRMR, which specifically evaluates reasoning-intensive retrieval. With the same Qwen3-VL-4B embedder backbone, ReWAM scores 53.3 versus 46.8 for Embed-RL, an absolute gain of 6.5 points (Table 2). It also outperforms Embed-RL on all four Theorem subtasks, with gains of 7.3, 6.6, 8.0, and 3.4 points in Math, Physics, Engineering, and Business, respectively.

Relevant MRMR scores from Table 2 are summarized below. Both methods use the Qwen3-VL-4B embedder backbone and full CoTs without RAI.

MethodMathPhy.Eng.Bus.Overall
Embed-RL32.945.635.152.646.8
ReWAM40.252.243.156.053.3
Gain (points)+7.3+6.6+8.0+3.4+6.5

Taken together, these results show that ReWAM's improvements over Embed-RL are not merely marginal, particularly on reasoning-intensive retrieval.

AuthorsOpenReview ↗

Response to Reviewer gdfg (2/4): W3


W3. Ablation Comparison with GRPO

Some major contributions have marginal effects. For instance, the proposed RASD has maximumly 0.5% improvement over GRPO (Table 4), the proposed CEPI also has 0.5% improvement over baseline (table 5).

3、Response.

We thank the reviewer for examining these ablations. We clarify that CEPI supplies the privileged guidance used by RASD. Table 4 evaluates token-level modulation over trajectory-level GRPO, while Table 5 adds CEPI components cumulatively to the same GRPO baseline. These are complementary analyses of the same training mechanism.

On MMEB-V2, the w/o RASD baseline scores 65.7 (Table 3). With the same backbones and RL training setup, trajectory-level GRPO reaches 66.0, whereas RASD reaches 66.5 (Table 4). Thus, GRPO adds 0.3 points over the baseline, and RASD adds a further 0.5 points over GRPO. This demonstrates the additional value of token-level guidance on the challenging 78-dataset general-purpose retrieval benchmark.

RASD improves retrieval reasoning by reinforcing input-supported, discriminative evidence and discouraging unsupported claims. MMEB-V2's general-purpose average does not specifically isolate these reasoning improvements. We have therefore added a GRPO-versus-RASD ablation on the reasoning-intensive MRMR benchmark in Table 2. With RAI disabled for both variants, RASD achieves larger gains of 1.5 and 2.2 points with the 4B and 7B embedders, respectively.

EmbedderTrainingKnow.Theo.Contr.OverallGain
Qwen3-VL-4BRASD71.947.935.653.3+1.5
Qwen3-VL-4BGRPO70.846.234.051.8—
Qwen2-VL-7BRASD70.243.337.151.4+2.2
Qwen2-VL-7BGRPO68.341.334.249.2—

Domain averages use unrounded subtask scores; Overall averages all 11 subtasks. These additional ablations provide clear evidence that RASD improves retrieval performance over standard trajectory-level GRPO, with consistent gains at both embedder scales and both benchmarks.

AuthorsOpenReview ↗

Response to Reviewer gdfg (3/4): W4


W4. Fairness of inference-speed comparisons

The inference speed comparison is not totally fair. First, this paper uses vLLM to accelerate generation, while previous works like Embed-RL did not. Second, it uses speculative decoding, which is a standard technique in inference acceleration and cannot be seen as a contribution here. Third, it uses a batch size of 1 for inference, which is advantageous to its framework with separate reasoner and embedder (compared to e.g. PLUME, UME-R1).

4、Response.

We thank the reviewer for raising these concerns. Our inference-speed comparison is fair under the common evaluation setting used for all methods, as detailed in §4.1 and Appendix B.4. We address the three points below.

  1. vLLM is used for quality evaluation, not speed measurement. MMEB-V2 and MRMR contain large collections of queries and retrieval candidates. Generating CoTs for these inputs without an optimized inference framework would make full-benchmark evaluation slow. We therefore use vLLM for CoT generation in the quality evaluations in Tables 1 and 2. The throughput measurements do not use vLLM. All compared methods are re-benchmarked on a single H800 GPU at batch size 1. Timing covers each method's complete encoding pipeline, including all reasoning steps and embedding extraction. We have clarified this separation in §4.1 and Appendix B.4.

  2. System contribution of RAI. We agree that speculative decoding is an established technique. We do not propose a new speculative decoding algorithm. We contribute an acceleration system for reasoning-enhanced UME that combines speculative decoding with retrieval-aware stopping. The two components address complementary sources of cost. Speculative decoding accelerates token generation, while the Retrieval Utility Head stops continuations with little predicted retrieval benefit.

    Baseline uses the RASD-trained reasoner with autoregressive decoding and without RAI. The accelerated variants use speculative decoding alone or with the Retrieval Utility Head. All variants use the same reasoner and Qwen2-VL-2B embedder. Both accelerated variants use K=7. Throughput and absolute gains are in samples/s (Table 6):

    ModalityBaselineSpeculative decodingGainWith utility headExtra gainGain ratio
    Image1.22.9+1.74.5+1.694.1%
    VisDoc1.12.3+1.23.3+1.083.3%
    Video1.12.5+1.43.4+0.964.3%

    Speculative decoding adds 1.2–1.7 samples/s, and the head adds a further 0.9–1.6 samples/s. The head's added gain is therefore 64.3–94.1% of the gain from speculative decoding alone. This controlled comparison shows a substantial contribution beyond the existing acceleration technique.

  3. Why we evaluate inference speed with batch size 1. First, we follow the batch-size-1 setting used by PLUME (ACMMM 2026) and apply it to all methods in our speed comparison. Second, we measure throughput for the full decoupled pipeline, including both CoT generation and embedding extraction. The additional cost of the separate reasoner and embedder is fully counted (Appendix B.4), so ReWAM receives no advantage from excluding either stage. Moreover, Table 6 compares RAI variants with the same reasoner and embedder backbones at batch size 1, showing acceleration with the backbone configuration held fixed. Finally, several baseline implementations have not been adapted to optimized inference frameworks for batched serving. Larger-batch comparisons would therefore mix differences in model design with differences in inference-engine adaptation. Using batch size 1 reduces this confound and supports a more direct comparison of the complete encoding costs across frameworks.

AuthorsOpenReview ↗

Response to Reviewer gdfg (4/4): Q1–Q2


Questions

Q1. Early stopping through RL length rewards

Regarding the RAI, could we not integrate the "early stopping" into the RL stage? E.g. to reward shorter sequences. Training an extra early stopping gate requires one extra stage, and will always impair performance, as the frozen embedder is trained with full sequences.

5、Response.

We thank the reviewer for this suggestion and apologize for not discussing it more explicitly. Combining retrieval rewards with a length penalty during RL is a promising way to learn more efficient reasoning. Such a joint objective could encourage shorter CoTs that retain useful retrieval evidence and allow the reasoner to learn when to stop without a separate head. In our current design, RAI predicts the retrieval gains remaining beyond a CoT prefix and adjusts the stopping point for each input. RAI makes these stopping decisions without candidate comparisons or repeated embedder calls at inference. Fitting the additional utility head takes less than one hour on a single GPU, so this stage requires limited resources and no further RL training. Although RAI at K=7 incurs a slight 0.4-point score drop, it increases throughput from 1.1 to 3.7 samples/s, offering a favorable accuracy-speed trade-off. We have added a discussion of this joint RL approach as future work in Appendix E.


Q2. Throughput/QPS comparisons

Could you also show the throughput/QPS results and compare with e.g. UME-R1, PLUME, TTE-Flash etc.?

6、Response.

We thank the reviewer for this suggestion. Fig. 1(e) already reports end-to-end encoding throughput, which also represents QPS under our evaluation protocol. All methods are evaluated on a single H800 GPU with batch size 1, and the exact values are listed below.

MethodEmbedderThroughput / QPS (samples/s)
UME-R1Qwen2-VL-2B0.12
Embed-RLQwen3-VL-2B0.75
RIMEQwen2-VL-2B0.94
PLUMEQwen2-VL-2B3.41
ReWAM (RAI, K=7)Qwen2-VL-2B3.75

The reported throughput covers the complete encoding pipeline, including all reasoning steps and embedding extraction. TTE-Flash is not included in our same-machine throughput comparison because its code is not publicly available, preventing us from re-benchmarking it under the same protocol. We apologize for overlooking this work in Related Work and have added it to §2 of the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.