acceptodds
Under review as a conference paper at ICLR 2027

Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings

10 views

Abstract

Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior–posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Retrieval, Multimodal Language Models, Latent Variable

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J7JzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PILOT, a framework that uses training-time relational information to improve reasoning exploration for multimodal embeddings. A shared MLLM implements an input-only prior and a posterior that additionally observes a randomly ordered positive–negative candidate pair with relevance labels withheld. Following dual-mode SFT initialization, the framework jointly optimizes both modes and selectively transfers useful posterior trajectories to the prior through importance-weighted fitting. At inference time, only the prior is used, preserving independent query and candidate encoding. The method achieves notable performance improvements on MMEB-V2.

Critical strengths
  1. Clear motivation and a well-motivated method. The approach uses training-time positive and negative candidates to help discover discriminative cues that input-only reasoning may overlook, then selectively transfers this capability to inference-time reasoning. The rationale for the individual components is intuitive, and the overall approach is appealing.

  2. Comprehensive ablations and deeper analysis of reasoning traces. The experiments examine initialization, joint optimization, and the transfer mechanism. The additional analyses investigate reasoning quality, posterior exploration, and the transfer of useful trajectories to the prior.

Critical weaknesses
  1. The analysis in Figure 3 remains somewhat coarse-grained. While I appreciate the authors’ efforts to analyze reasoning quality and leakage, both may vary with dataset source, task type, and question difficulty, and the current analysis does not sufficiently disentangle these factors. Furthermore, quality and leakage are assessed by the same judge model, which may introduce correlated evaluation biases. More fine-grained stratification and cross-checks with another judge model or a small human audit would strengthen the conclusions. The leakage metric also combines candidate-only factual content with explicit candidate-comparison narration; reporting these separately would make the results easier to interpret.

  2. Presentation issues. Figures 1 and 2 could benefit from further refinement. Some elements are currently somewhat ambiguous, and the figures give the impression of AI-generated flowcharts that have not been sufficiently polished. Improving their clarity and precision would help communicate the method more effectively.

Questions for authors

Could the authors clarify the actual value and sign convention of β\beta in the importance weights? The main text uses exp⁡(βRret)\exp(\beta R_{\mathrm{ret}}), while the appendix specifies β=−10\beta=-10 and states that higher RretR_{\mathrm{ret}} indicates better retrieval performance. Is this due to a different reward/loss convention, or is it a typographical error? Additionally, does the denominator in Equation (15) include the positive term?

3 comments

Thank you for suggesting a more fine-grained analysis of reasoning quality and leakage, for recommending clearer figures, and for flagging two points that need clarification: the sign convention for β\beta in the importance weights and the denominator in Eq. (15).

Q1: The analysis in Figure 3 remains somewhat coarse-grained. While I appreciate the authors’ efforts to analyze reasoning quality and leakage, both may vary with dataset source, task type, and question difficulty, and the current analysis does not sufficiently disentangle these factors. Furthermore, quality and leakage are assessed by the same judge model, which may introduce correlated evaluation biases. More fine-grained stratification and cross-checks with another judge model or a small human audit would strengthen the conclusions. The leakage metric also combines candidate-only factual content with explicit candidate-comparison narration; reporting these separately would make the results easier to interpret.

Answer: Fig. 3 uses 1,000 queries drawn from all 78 MMEB-V2 subsets, with 12 or 13 queries per subset. This gives each source a comparable representation, but the domain averages can still mask differences across tasks and difficulty levels. In the revision, we will report the judged reasoning quality and leakage by task type and by query difficulty, measured using the rank of the highest-ranked positive under the input-only SFT baseline.

We also tested one aspect of difficulty that we can vary while holding the query fixed: how difficult the negative candidate is to distinguish from the positive. For each probe query, we compared the highest-ranked candidate outside the known positives under the 2B dual-mode SFT checkpoint with a random negative from the same subset. We held the checkpoint, query, positive candidate, prompt, and decoding settings fixed. In each condition, we sampled 16 posterior traces per query and replayed each trace under the input-only prompt to extract its embedding. Both conditions used identical retrieval pools and candidate embeddings. Pass@16 is reported in this table:

DomainQueriesHard-negative posteriorRandom-negative posteriorRandom − Hard
Overall1,00058.50%68.10%+9.60 pp
Image44769.35%76.06%+6.71 pp
Video22053.64%60.91%+7.27 pp
VisDoc33347.15%62.16%+15.02 pp

Random negatives yielded higher Pass@16 in all three domains, with the largest gain on VisDoc (+15.02 percentage points). This experiment shows that negative selection affects the posterior’s ability to generate at least one retrieval-useful trace. The task- and difficulty-level analysis above will examine how judged quality and leakage vary across queries.

To check for correlated judge bias, we will include a small human audit stratified by domain and task type. Reviewers will independently assess reasoning quality and leakage without seeing the model’s ratings, and we will report human–judge agreement for each. We will also report separate rates for candidate-only factual content and explicit candidate-comparison narration, along with their overlap, to clarify what contributes to the 79.6% leakage rate.

Q2: Presentation issues. Figures 1 and 2 could benefit from further refinement. Some elements are currently somewhat ambiguous, and the figures give the impression of AI-generated flowcharts that have not been sufficiently polished. Improving their clarity and precision would help communicate the method more effectively.

Answer: We will revise Figs. 1 and 2 to make the distinction between prior and posterior reasoning clearer. We will label the inputs available to each mode, show where posterior traces guide the prior during training, and make the prior-only inference path explicit. We will also simplify the layouts and use consistent labels and arrows across both figures.

Q3: Could the authors clarify the actual value and sign convention of β\beta in the importance weights? The main text uses exp⁡(βRret)\exp(\beta R_{\mathrm{ret}}), while the appendix specifies β=−10\beta=-10 and states that higher RretR_{\mathrm{ret}} indicates better retrieval performance. Is this due to a different reward/loss convention, or is it a typographical error? Additionally, does the denominator in Equation (15) include the positive term? Anwer: Thank you for spotting this!. Both points are errors in the manuscript, but neither affects the reported results.

Sign and value of β\beta. The minus sign in the appendix is a typo and not a different convention. Throughout the paper RretR_{\mathrm{ret}} is a reward for which higher is better, and β\beta is a positive inverse temperature, so the importance weight exp⁡(βRret)\exp(\beta R_{\mathrm{ret}}) increases with retrieval quality, as written in the main text. The value used in all experiments is β=10\beta=10. A negative β\beta would up-weight the posterior traces that retrieve worst, which is the opposite of what our implementation does. The weights are normalized over the posterior traces sampled for the same query, wk=exp⁡(βRret(eθ(x,zk),R))∑j=1Kexp⁡(βRret(eθ(x,zj),R)).w_k = \frac{\exp\big(\beta R_{\mathrm{ret}}(\mathbf{e}_{\theta}(x,z_k),\mathcal{R})\big)}{\sum_{j=1}^{K} \exp\big(\beta R_{\mathrm{ret}}(\mathbf{e}_{\theta}(x,z_j),\mathcal{R})\big)}. so only reward differences within a query affect them. We will correct it in the appendix.

Denominator of Eq. (15). Thank you for catching this. The denominator as printed sums over the negatives only, which is an error in the manuscript. In our implementation, the denominator also includes the positive term, so that RretR_{\mathrm{ret}} is the log-probability of the positive under a softmax over the positive and the in-batch negatives. The corrected equation is

Rret(eθ(x,zk),R)=log⁡∑m=1Mexp⁡(sk,m+/T)∑m=1Mexp⁡(sk,m+/T)+∑x−∈X−∑m=1Mexp⁡(sk,m−/T),R_{\mathrm{ret}}(\mathbf{e}_{\theta}(x, z_k), \mathcal{R}) = \log \frac{\sum_{m=1}^{M} \exp\big(s^{+}_{k,m} / T\big)}{\sum_{m=1}^{M} \exp\big(s^{+}_{k,m} / T\big) + \sum_{x^{-} \in \mathcal{X}^{-}} \sum_{m=1}^{M} \exp\big(s^{-}_{k,m} / T\big)},

where sk,m+=sim(eθ(x,zk),eθ(x+,zm))s^{+}_{k,m} = \mathrm{sim}\big(\mathbf{e}_{\theta}(x, z_k), \mathbf{e}_{\theta}(x^{+}, z_m)\big) and sk,m−=sim(eθ(x,zk),eθ(x−,zm))s^{-}_{k,m} = \mathrm{sim}\big(\mathbf{e}_{\theta}(x, z_k), \mathbf{e}_{\theta}(x^{-}, z_m)\big). Including the positive term in the denominator ensures that Rret≤0R_{\mathrm{ret}} \le 0. The reward approaches zero as the positive term dominates the sum of the negative terms. Omitting the positive term, as in the printed equation, removes this upper bound. Our implementation included the positive term in all reported experiments, so the correction affects only the equation as printed.

Thank you for the clarification and the additional experiments, which address my concerns. I will maintain my positive score of 3. Please also improve the clarity and readability of Figures 1 and 2 to enhance the overall presentation.

Reviewer_Ctm1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PILOT, which employs a privileged posterior mode during the training phase and an input-only prior mode during the inference phase. The model is initialized via SFT on synthetic reasoning trajectories, followed by joint evolution using the GRPO algorithm. PILOT outperforms UME-R1 across all 12 task groups.

Critical strengths
  1. Leverage relational evidence during the training phase to guide the exploration of reasoning, rather than merely using it to score outcomes post-hoc. This approach is both well-founded and principled.

  2. PILOT consistently outperforms UME-R1. Ablation studies further demonstrate that the gains derived from posterior information are not simply a result of an increased number of samples.

Critical weaknesses

The results show that, in 79.6% of cases, blind posterior traces leak information pertaining solely to the candidate solutions. Hence, the performance gains could from the information leakage rather than from more effective reasoning exploration.

Questions for authors

How much of the performance gain provided by VisDoc remains after excluding leaked posterior traces

Could the authors provide the standard deviations for the ablation study results shown in Table 2

2 comments

Thank you for raising the question of how candidate information in posterior traces may affect retrieval and for asking us to report variability across seeds.

Q1: “The results show that, in 79.6% of cases, blind posterior traces leak information pertaining solely to the candidate solutions. Hence, the performance gains could from the information leakage rather than from more effective reasoning exploration.” “How much of the performance gain provided by VisDoc remains after excluding leaked posterior traces.”

Answer: As Fig 2 shows, PILOT uses candidate-conditioned posterior traces only during training. At inference, the prior generates each reasoning trace from the target input alone. This rules out direct access to privileged candidate information when computing the Table 1 results. The reviewer’s concern still applies to training, where posterior traces guide the prior during co-evolution. A posterior can copy a discriminative detail from a candidate even when the query provides no basis for inferring it. Such a trace may score well for retrieval yet provide poor supervision for the query-only prior. This motivates the importance weighting in LtransferL_{\mathrm{transfer}}, which gives less weight to traces that the prior is unlikely to generate, even when their retrieval utility is high.

Section 4.4 (RQ1) examines how often this occurs. We used GPT-5.6 sol to judge retrieval usefulness and separately flag candidate-only information. Blind-contrastive prompting had the highest leakage rate: 79.6% of its traces were flagged across the full probe (Fig. 3A). Yet on the shared subset of queries where no prompt variant was flagged, blind-contrastive traces still achieved the highest judged retrieval-usefulness margin (0.395, compared with 0.275 for positive-only and 0.204 for labeled-pair prompting; Fig. 3C). These findings suggest that comparing candidates can both encourage copying and help the annotator identify discriminative details grounded in the query. To assess downstream retrieval, we grouped the 333 VisDoc probe queries by whether the judge flagged their blind posterior trace. Both models were evaluated using only the inputs available at inference:

VisDoc subsetQueriesUME-R1PILOTDifference95% bootstrap CI
Leaking queries25766.56766.211−0.356[−2.006, +1.361]
Non-leaking queries7664.52466.923+2.400[0.000, +7.200]

After excluding evaluation queries with flagged traces, PILOT’s observed advantage over UME-R1 was 2.400 nDCG@5 points. The confidence interval reaches zero, so the size of this advantage remains uncertain. Measuring how much gain survives removing flagged traces from training would require auditing the 676K posterior training examples and retraining PILOT. That experiment was beyond our compute budget.

Q2: Could the authors provide the standard deviations for the ablation study results shown in Table 2

Answer: We report the mean ± standard deviation across those seeds for each modality below:

VariantSFT initializationPrior GRPOPosterior during RLIW prior fittingRollouts/query (prior + posterior)ImageVideoVisDocAll
PILOTDual-mode✓GRPO-updated✓8 + 867.5 (±\pm 0.17)45.9 (±0.08\pm 0.08)69.9 (±0.20\pm 0.20)63.3
Joint GRPODual-mode✓GRPO-updated×8 + 866.8 (±0.09\pm 0.09)45.1 (±0.12\pm 0.12)68.1 (±0.14\pm 0.14)62.2 (−1.1)
Frozen-posterior variantDual-mode✓Frozen×8 + 067.1 (±0.12\pm 0.12)44.8 (±0.23\pm 0.23)66.1 (±0.18\pm 0.18)61.6 (−1.7)
Prior GRPO onlyDual-mode✓Not used×16 + 066.5 (±0.16\pm 0.16)43.4 (±0.21\pm 0.21)67.2 (±0.07\pm 0.07)61.4 (−1.9)
Dual-mode SFTDual-mode————65.7 (±0.11\pm 0.11)42.1 (±0.09\pm 0.09)65.9 (±0.10\pm 0.10)60.3 (−3.0)
Direct DistillationPrior————64.7 (±0.14\pm 0.14)41.6 (±0.17\pm 0.17)64.8 (±0.15\pm 0.15)59.4 (−3.9)
Prior SFTPrior————65.3 (±0.21\pm 0.21)41.0 (±0.09\pm 0.09)63.2 (±0.13\pm 0.13)59.0 (−4.3)
Reviewer_6bVkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses an exploration bottleneck in reasoning‑augmented multimodal embedding models. Existing methods generate reasoning traces only conditioned on individual inputs and use pairwise retrieval relevance merely as post‑hoc reward; rewards cannot discover discriminative reasoning trajectories that have never been sampled. The authors propose PILOT, a latent‑variable framework which treats reasoning traces as latent variables.

Critical strengths
  1. This paper identifies the exploration bottleneck, i.e., post‑hoc rewards cannot produce unseen discriminative reasoning.

  2. Instead of naive knowledge distillation, PILOT transfers trajectories weighted by both retrieval utility and prior reachability.

  3. Comprehensive ablations and diagnostic probing validate each key component and mechanism.

Critical weaknesses
  1. The whole pipeline proposed in this paper relies on a stronger separate MLLM to generate high‑quality prior/blind‑contrast posterior traces.

  2. The paper does not sufficiently analyse performance sensitivity to annotator capability. It remains unclear how much performance would degrade if a weaker annotator was used. This problem limits easy reproduction for labs without access to capable multimodal annotator models.

  3. All main experiments are built on Qwen2‑VL‑2B/7B. We cannot confirm whether the observed performance gains of prior‑posterior co‑evolution will transfer to other multimodal LLM backbones.

Questions for authors
  1. How would the performance change if you use a weaker multimodal model for trace synthesis?

  2. This paper uses Qwen2‑VL family only. Could you discuss what obstacles exist to porting PILOT to other MLLM backbones (or latest Qwen-VL), and share your expectation whether similar gains can be reproduced?

1 comment

Thank you for the thoughtful questions about how PILOT’s performance depends on the choice of annotator and backbone.

Q1: “The whole pipeline proposed in this paper relies on a stronger separate MLLM to generate high‑quality prior/blind‑contrast posterior traces.” “The paper does not sufficiently analyse performance sensitivity to annotator capability. It remains unclear how much performance would degrade if a weaker annotator was used. This problem limits easy reproduction for labs without access to capable multimodal annotator models.” “How would the performance change if you use a weaker multimodal model for trace synthesis?”

Answer: We use GLM-4.1V-9B-Thinking to match UME-R1’s annotator. We reuse UME-R1’s released prior traces and use the 9B model offline only to synthesize posterior traces. To quantify sensitivity to annotator choice, we will compare a 4B model, the current 9B model, and a larger model using the same training pairs, prompts, verification criteria, and SFT settings. We will report verification pass rates and downstream retrieval performance for each.

Section 4.4 provides some insight into which aspects of annotation quality may matter. The blind-contrastive prompt asks the annotator to use the candidates to identify discriminative details that can be grounded in the query itself. Although the judge flagged candidate-only information in 79.6% of blind-contrastive traces, blind-contrastive prompting also achieves the highest judged retrieval-usefulness margin on the shared subset where no leakage is flagged under any prompt (0.395, compared with 0.275 for positive-only and 0.204 for labeled-pair prompting). This suggests that candidate context can help the annotator identify useful, query-grounded details. Our hypothesis is that annotators with stronger reasoning and instruction-following abilities can identify such details more reliably. The proposed ablation will test whether this translates into better retrieval after SFT.

Q2: “All main experiments are built on Qwen2‑VL‑2B/7B. We cannot confirm whether the observed performance gains of prior‑posterior co‑evolution will transfer to other multimodal LLM backbones.” “This paper uses Qwen2‑VL family only. Could you discuss what obstacles exist to porting PILOT to other MLLM backbones (or latest Qwen-VL), and share your expectation whether similar gains can be reproduced?”

Answer: We chose Qwen2-VL-2B/7B to enable a controlled comparison with our main baseline (UME-R1), which uses the same backbones and training data. Reproducing all baselines on a newer backbone would exceed our compute budget by far. In addition, some baselines (e.g., RzenEmbed) have not released their full training data.

PILOT’s training objective does not depend on Qwen2-VL-specific features. PILOT improves over UME-R1 by 3.2 points at 2B and 3.3 points at 7B on MMEB-V2 (Table 1). More directly, the 2B ablation shows that, from the same dual-mode SFT checkpoint and with the same budget of 16 rollouts per query, co-evolution outperforms prior-only GRPO by 1.9 points (63.3 vs. 61.4; Table 2). Section 4.4 also shows that candidate-conditioned reasoning finds successful retrieval traces more often than query-only reasoning, even when the prior receives twice as many samples. Based on these results, we predict that co-evolution will improve retrieval on other backbones when candidate context helps the posterior discover useful traces that the prior can learn to generate. The size of the gain may vary, particularly if a stronger backbone already produces effective query-only traces. To test this directly, we will evaluate Qwen3-VL before and after co-evolution using the same SFT checkpoint and retrieval benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.