- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PILOT, a framework that uses training-time relational information to improve reasoning exploration for multimodal embeddings. A shared MLLM implements an input-only prior and a posterior that additionally observes a randomly ordered positive–negative candidate pair with relevance labels withheld. Following dual-mode SFT initialization, the framework jointly optimizes both modes and selectively transfers useful posterior trajectories to the prior through importance-weighted fitting. At inference time, only the prior is used, preserving independent query and candidate encoding. The method achieves notable performance improvements on MMEB-V2.
Critical strengths
-
Clear motivation and a well-motivated method. The approach uses training-time positive and negative candidates to help discover discriminative cues that input-only reasoning may overlook, then selectively transfers this capability to inference-time reasoning. The rationale for the individual components is intuitive, and the overall approach is appealing.
-
Comprehensive ablations and deeper analysis of reasoning traces. The experiments examine initialization, joint optimization, and the transfer mechanism. The additional analyses investigate reasoning quality, posterior exploration, and the transfer of useful trajectories to the prior.
Critical weaknesses
-
The analysis in Figure 3 remains somewhat coarse-grained. While I appreciate the authors’ efforts to analyze reasoning quality and leakage, both may vary with dataset source, task type, and question difficulty, and the current analysis does not sufficiently disentangle these factors. Furthermore, quality and leakage are assessed by the same judge model, which may introduce correlated evaluation biases. More fine-grained stratification and cross-checks with another judge model or a small human audit would strengthen the conclusions. The leakage metric also combines candidate-only factual content with explicit candidate-comparison narration; reporting these separately would make the results easier to interpret.
-
Presentation issues. Figures 1 and 2 could benefit from further refinement. Some elements are currently somewhat ambiguous, and the figures give the impression of AI-generated flowcharts that have not been sufficiently polished. Improving their clarity and precision would help communicate the method more effectively.
Questions for authors
Could the authors clarify the actual value and sign convention of in the importance weights? The main text uses , while the appendix specifies and states that higher indicates better retrieval performance. Is this due to a different reward/loss convention, or is it a typographical error? Additionally, does the denominator in Equation (15) include the positive term?
3 comments
Q3: Could the authors clarify the actual value and sign convention of in the importance weights? The main text uses , while the appendix specifies and states that higher indicates better retrieval performance. Is this due to a different reward/loss convention, or is it a typographical error? Additionally, does the denominator in Equation (15) include the positive term? Anwer: Thank you for spotting this!. Both points are errors in the manuscript, but neither affects the reported results.
Sign and value of . The minus sign in the appendix is a typo and not a different convention. Throughout the paper is a reward for which higher is better, and is a positive inverse temperature, so the importance weight increases with retrieval quality, as written in the main text. The value used in all experiments is . A negative would up-weight the posterior traces that retrieve worst, which is the opposite of what our implementation does. The weights are normalized over the posterior traces sampled for the same query, so only reward differences within a query affect them. We will correct it in the appendix.
Denominator of Eq. (15). Thank you for catching this. The denominator as printed sums over the negatives only, which is an error in the manuscript. In our implementation, the denominator also includes the positive term, so that is the log-probability of the positive under a softmax over the positive and the in-batch negatives. The corrected equation is
where and . Including the positive term in the denominator ensures that . The reward approaches zero as the positive term dominates the sum of the negative terms. Omitting the positive term, as in the printed equation, removes this upper bound. Our implementation included the positive term in all reported experiments, so the correction affects only the equation as printed.
Thank you for the clarification and the additional experiments, which address my concerns. I will maintain my positive score of 3. Please also improve the clarity and readability of Figures 1 and 2 to enhance the overall presentation.
Thank you for suggesting a more fine-grained analysis of reasoning quality and leakage, for recommending clearer figures, and for flagging two points that need clarification: the sign convention for β in the importance weights and the denominator in Eq. (15).
Q1: The analysis in Figure 3 remains somewhat coarse-grained. While I appreciate the authors’ efforts to analyze reasoning quality and leakage, both may vary with dataset source, task type, and question difficulty, and the current analysis does not sufficiently disentangle these factors. Furthermore, quality and leakage are assessed by the same judge model, which may introduce correlated evaluation biases. More fine-grained stratification and cross-checks with another judge model or a small human audit would strengthen the conclusions. The leakage metric also combines candidate-only factual content with explicit candidate-comparison narration; reporting these separately would make the results easier to interpret.
Answer: Fig. 3 uses 1,000 queries drawn from all 78 MMEB-V2 subsets, with 12 or 13 queries per subset. This gives each source a comparable representation, but the domain averages can still mask differences across tasks and difficulty levels. In the revision, we will report the judged reasoning quality and leakage by task type and by query difficulty, measured using the rank of the highest-ranked positive under the input-only SFT baseline.
We also tested one aspect of difficulty that we can vary while holding the query fixed: how difficult the negative candidate is to distinguish from the positive. For each probe query, we compared the highest-ranked candidate outside the known positives under the 2B dual-mode SFT checkpoint with a random negative from the same subset. We held the checkpoint, query, positive candidate, prompt, and decoding settings fixed. In each condition, we sampled 16 posterior traces per query and replayed each trace under the input-only prompt to extract its embedding. Both conditions used identical retrieval pools and candidate embeddings. Pass@16 is reported in this table:
Random negatives yielded higher Pass@16 in all three domains, with the largest gain on VisDoc (+15.02 percentage points). This experiment shows that negative selection affects the posterior’s ability to generate at least one retrieval-useful trace. The task- and difficulty-level analysis above will examine how judged quality and leakage vary across queries.
To check for correlated judge bias, we will include a small human audit stratified by domain and task type. Reviewers will independently assess reasoning quality and leakage without seeing the model’s ratings, and we will report human–judge agreement for each. We will also report separate rates for candidate-only factual content and explicit candidate-comparison narration, along with their overlap, to clarify what contributes to the 79.6% leakage rate.
Q2: Presentation issues. Figures 1 and 2 could benefit from further refinement. Some elements are currently somewhat ambiguous, and the figures give the impression of AI-generated flowcharts that have not been sufficiently polished. Improving their clarity and precision would help communicate the method more effectively.
Answer: We will revise Figs. 1 and 2 to make the distinction between prior and posterior reasoning clearer. We will label the inputs available to each mode, show where posterior traces guide the prior during training, and make the prior-only inference path explicit. We will also simplify the layouts and use consistent labels and arrows across both figures.