acceptodds
Withdrawn from ICLR 2027

Closing the Trajectory Gap: On-policy Learning For Multimodal Reasoning Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Universal multimodal embeddings support a broad range of applications through a shared representation space. Although they build on multimodal large language models, most existing methods use these models primarily as encoders, leaving their generative capabilities underexplored. Recent reasoning-enhanced approaches generate intermediate reasoning before producing the final embedding and have shown promising performance on multimodal embedding benchmarks. However, training a unified reasoner–embedder often relies on task-specific chains of thought generated and filtered offline, while jointly optimizing language modeling and contrastive learning on these fixed trajectories. Beyond the cost of constructing this offline supervision, the model is trained on externally generated trajectories but conditions its inference-time embeddings on its own generations, creating a reasoning trajectory gap. Our controlled experiments suggest that this gap can undermine retrieval performance. We introduce OPR-Embed (On-policy Reasoning Embedding), a two-stage on-policy framework for multimodal reasoning embeddings. Without any pre-built CoT corpus, Stage I performs embedding-attentive on-policy distillation by scoring prefixes sampled from the current student with a frozen teacher and weighting token-level supervision using attention from the final embedding readout. Stage II uses verifier-guided group relative policy optimization to refine complete trajectories using retrieval feedback together with a candidate-match verifier. Experiments on MMEB-v2 demonstrate that OPR-Embed is competitive with state-of-the-art methods, achieving overall scores of 65.8 and 70.0 with 2B and 7B backbones, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
MLLM Reasoning, MLLM Embeddings, On-policy Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CNRtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OPR-EMBED, which is a reasoning-enhanced UME model with a single MLLM serving as both the reasoner and student. The main contribution is to replace the existing SFT stage with on-policy distillation (OPD), followed by the GRPO stage similar to UME-R1.

Critical strengths

I don't feel there are critical strengths from the paper.

Critical weaknesses
  1. Novelty. The biggest contribution claimed by the paper is the data-free OPD stage, which replaces SFT. However I do not feel this is particularly novel. Besides OPD, the GRPO stage seems to be directly adopted from UME-R1.
  2. Limited gain. In table 1 OPR-EMBED barely outperforms MMEmb-R1 (+0.008), and this margin can be even smaller if we consider the two eval sets bugs which is fixed in this work but not in the previous ones (https://github.com/TIGER-AI-Lab/VLM2Vec/issues/167). Each component's gain is also marginal. For instance, the proposed attentive OPSD vs OPSD has 0.005 improvement, with GRPO it's +0.001, and with the proposed verifier it's +0.002. These marginal improvements do not justify the design.
  3. Limited ablations. There is no ablation on the OPD's effect on the final accuracy after GRPO stage, and little ablations on each design choices of OPD stage. Also, there is no analysis on failure cases (for instance, it seems that the model performs particularly poor on video tasks compared to previous works).
Questions for authors

Please see weaknesses.

1 comment

Thank you for reading our paper and for your comments and suggestions. We address your concerns below.

W1. Novelty and contribution

We respectfully disagree with the characterization of our work as merely "replacing SFT with OPD, followed by GRPO." We first identify and analyze the trajectory gap in joint training, then develop a training scheme around this diagnosis, which is a central empirical contribution of the paper.

Figure 1 shows that Oracle-CoT SFT achieves 54.0 with self-generated reasoning, below Self-CoT SFT's 61.3, despite the stronger annotation model and verification used for Oracle CoTs. This comparison holds initialization, training pairs, the joint objective, and optimization settings fixed. Supplying target-assisted Oracle trajectories to the same Oracle-SFT checkpoint raises the score to 78.1 without updating parameters or changing the encoder and embedding readout. This separates the ability to exploit supplied reasoning from the ability to generate reasoning that supports retrieval. Appendix B.3 provides complementary evidence: the Oracle-trained model has a larger train-to-self NLL gap than the Self-CoT-trained model (0.58 versus 0.08 nats).

To address this gap, we jointly supervise reasoning and embeddings on the same current-student trajectories. The teacher evaluates the student's own prefixes rather than providing a fixed corpus to imitate, while embedding-readout attention weights token-level supervision toward the final representation. This connects the diagnosis to the training design and removes the separate offline CoT generation and curation pipeline.

Stage II also goes beyond directly adopting UME-R1: building on GRPO and retrieval rewards, we introduce candidate-match feedback to adjust trajectory advantages and withhold positive credit from mismatched trajectories (Eqs. 8-9). Retrieval rewards assess embedding rankings but do not directly evaluate rationale-target matching. The verifier addresses this distinction, and the constraint is applied to advantages because penalizing rewards alone need not prevent a rejected trajectory from receiving positive group-relative credit. Together, the two stages tailor token-level supervision and trajectory-level credit assignment to retrieval.


W2. Performance gains and evaluation versions

The final margin over MMEmb-R1 and the incremental gains from subsequent components do not fully capture this work's contribution. Identifying and analyzing the trajectory gap has independent empirical value; our method experiments further show that competitive retrieval performance can be achieved without pre-built CoT annotations.

Against our own no-CoT control, Overall improves from 61.1 to 65.8. Basic OPSD alone reaches 65.0, accounting for the main gain; attention and verifier guidance further refine supervision. These components refine the core on-policy training scheme through retrieval-specific supervision. The practical result is a competitive reasoning embedder trained without an offline CoT annotation pipeline. The verifier is used only during training and adds no query-time inference cost.

Regarding the evaluation bug, the benchmark maintainers released the fixes on November 4, 2025, before MMEmb-R1 appeared on arxiv in April 2026. Because its code and checkpoints have not been released, we cannot independently verify which evaluation version it used. It is therefore not justified to assume that it used the old evaluation or to infer that our margin over it would shrink on that basis.


W3. Ablations and failure cases

Thank you for your suggestions on additional ablations and failure-case analysis.

Table 2 compares uniform and attentive OPSD, while Table 4 examines removing OPD, target access, and teacher capacity. Removing the OPD objective lowers Overall from 65.5 to 62.7 and produces 100% truncation, whereas removing target access retains 64.9. These results help distinguish the role of on-policy language supervision in maintaining generation from the additional guidance provided by the paired target.

We further tested target access with three seeds per setting: Overall is 65.01 +/- 0.06 without target access and 65.49 +/- 0.05 with it (mean +/- across-seed SD), supporting a stable additional gain of 0.48 points.

Regarding OPD's effect on final performance after GRPO, Table 2 reports continued GRPO training from attentive OPSD. We would appreciate your suggestions on where additional ablations would strengthen the evidence, and welcome further discussion. And we will add failure-case analyses in the revision.

Reviewer_7vJMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies joint reasoner–embedder training. Oracle-CoT SFT achieves only 54.0 Hit@1 with self-generated reasoning, but reaches 78.1 when supplied with target-assisted Oracle CoTs, suggesting a mismatch between training and inference trajectories. OPR-Embed first applies attentive on-policy distillation with a target-aware teacher, followed by GRPO with retrieval rewards and a 32B verifier, achieving 65.8/70.0 on MMEB-v2.

Critical strengths
  1. The same checkpoint improves from 54.0 to 78.1 with Oracle CoTs. Assuming no target leakage, this strongly supports the trajectory-gap hypothesis.
  2. The method avoids a pre-built offline CoT corpus and outperforms RGE-style Decoupled SFT by 2 points under the matched setting.
Critical weaknesses
  1. In Table 4, removing target access reduces performance by only 0.6 points, while removing the full OPD objective reduces it by 2.8 points and causes 100% truncation. This suggests that most gains come from on-policy language modeling rather than privileged target information, weakening the paper’s central narrative.
  2. Table 3 omits an anchor-based baseline following UME-R1 or RIME. Moreover, the claimed 7.8-point gain over VLM2Vec-V2 drops to 4.7 points against the paper’s own no-CoT baseline, suggesting that part of the headline improvement comes from the underlying training setup.
Questions for authors
  1. Please add a UME-R1/RIME-style anchor joint-SFT baseline under the matched 160K setting in Table 3 and compare it directly with Attentive OPSD.
  2. Is the 0.6-point gain from target access stable across random seeds? Please also conduct a human audit of the Oracle CoTs used for the 78.1 result and report the target-leakage rate.
1 comment

We sincerely appreciate your recognition of our findings and your thoughtful suggestions for strengthening the evidence.

W1: Sources of improvement and the central narrative.

We agree that most gains come from on-policy joint training, which aligns with our central narrative. Our starting point is the reasoning trajectory gap: being able to extract effective representations from annotated reasoning does not mean the model can generate retrieval-supporting reasoning itself. This motivates supervising reasoning and representations on the same current student trajectories. OPSD provides a simple, effective implementation without a prebuilt CoT corpus. Its substantial gains thus support the training approach motivated by our diagnosis.

In Table 4, removing target access lowers performance from 65.5 to 64.9; removing the entire OPD objective lowers it to 62.7 and causes 100% generation truncation. Our discussion of this table explicitly attributes the main gains to joint on-policy language modeling and contrastive learning. Target access gives the teacher additional paired-target information, providing further gains within this framework; the three-seed results below test their stability.

Attentive OPSD uses embedding readout attention to align token-level supervision with representation learning; verifier-guided GRPO combines retrieval feedback and candidate-matching judgments to improve trajectory-level supervision. Table 2 shows their respective contributions: basic OPSD improves the no-CoT baseline from 61.1 to 65.0; attentive weighting reaches 65.5, standard GRPO 65.6, and verifier-guided GRPO 65.8. These components further improve basic OPSD. Our central contribution remains identifying the importance of trajectory distributions in joint reasoning–representation training and developing an effective on-policy approach accordingly.


W2/Q1: Anchor baseline and the 7.8- vs. 4.7-point gains.

We added a UME-R1/RIME-style anchor joint-training baseline using Table 3's 160K training scale, Qwen2-VL-2B backbone, and MMEB-v1 evaluation setting. It adds contrastive supervision on representations extracted directly from the original query to Oracle-CoT SFT, while candidates remain directly encoded. All results below are Hit@1 (%); the other rows are from Table 3.

Training strategyCLSVQARETVGDINDOODOverall
No-CoT baseline57.157.263.480.263.659.561.8
Oracle-CoT SFT52.651.953.962.758.748.154.0
Oracle-CoT SFT + anchor58.156.662.181.164.757.661.6
Decoupled SFT61.962.464.282.668.460.965.1
Attentive OPSD66.062.566.882.970.263.467.1

The anchor baseline reaches 61.6, improving Oracle-CoT SFT by 7.6 points, but remains comparable to no-CoT (61.8) and below Decoupled SFT (65.1) and Attentive OPSD (67.1). Attentive OPSD outperforms anchor across all four task categories and both IND/OOD aggregates. Thus, adding anchor supervision helps but does not fully resolve the challenges of jointly training reasoning and representations.

Anchor directly supervises original-query representations, but its CoT-conditioned branch still relies on offline trajectories. To address information leakage, RGE-style Decoupled SFT separates the supervision paths, using Oracle CoT for language supervision and student-generated CoT for contrastive learning. We further address the training–inference trajectory gap by applying both forms of supervision to the same current student trajectories. Attentive OPSD achieves better retrieval performance without generating and curating an offline CoT corpus.

The 7.8-point gain is measured against VLM2Vec-V2. Against our own no-CoT baseline, OPR-Embed still improves performance from 61.1 to 65.8 (+4.7 points), demonstrating that the gains extend beyond differences in the underlying training setup.


Q2: Oracle-CoT audit and target-access stability.

For the Oracle CoTs used in the 78.1 Hit@1 result, we manually inspected 100 examples per dataset, totaling 2,000. We found no target leakage: the observed leakage rate in the audited sample was 0% (0/2,000).

We ran three random seeds each with and without target access. Below are MMEB-v2 means ± standard deviations across seeds; Overall aggregates all 78 datasets.

Target accessImageVideoVisDocOverall
No70.78 ± 0.0841.98 ± 0.1973.64 ± 0.1165.01 ± 0.06
Yes71.22 ± 0.0642.25 ± 0.2374.31 ± 0.1665.49 ± 0.05

Target access improves Image, Video, and VisDoc by 0.44, 0.27, and 0.67 points, respectively, and Overall by 0.48 (65.01 to 65.49). The Overall standard deviations, 0.06 and 0.05, are both smaller than the mean gain, supporting a stable additional benefit from target access.

Reviewer_BGn8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose OPR‑EMBED, a two‑stage fully on‑policy training framework that eliminates reliance on pre‑built CoT corpora.

Stage‑I is embedding‑attentive on‑policy self‑distillation: student samples reasoning trajectories on‑policy; a frozen teacher which has privileged access to paired target conducts token‑level distillation supervision weighted by embedding‑readout attention, jointly optimized with contrastive embedding loss.

Stage‑II applies verifier‑guided GRPO: multiple on‑policy rollouts are optimized with combined retrieval ranking reward plus a frozen‑LLM candidate‑match verifier to correct trajectory credit.

Critical strengths
  1. The controlled experiment clearly demonstrates that training on offline fixed CoT (even high‑quality oracle CoT) leads to performance degradation when the model must generate its own reasoning at test time.

  2. Embedding‑attentive on‑policy distillation weights distillation loss by embedding readout attention to prioritize tokens which matter for final retrieval embedding.

  3. The verifier‑guided GRPO further corrects trajectory credit: pure retrieval reward only measures embedding ranking quality and cannot tell whether reasoning itself is semantically valid; the candidate‑match verifier suppresses positive advantage for semantically invalid trajectories.

Critical weaknesses
  1. Stage‑II GRPO relies on a frozen Qwen3‑VL‑32B‑Instruct as candidate‑match verifier. The paper does not ablate how verifier model size / verifier noise impacts downstream retrieval performance.

  2. The method requires autoregressively generating reasoning text for every query during inference, adding latency compared with no‑reasoning embedding baselines.

  3. The paper shows strong evidence for trajectory mismatch but explicitly admits it may not be the only source of performance degradation for offline‑CoT joint training. It does not isolate or rule out other factors such as objective conflict between language modeling loss and contrastive embedding loss.

Questions for authors
  1. For the verifier in Stage‑II GRPO: Have you ablated verifier model capacity? How does retrieval performance change if you use a smaller, weaker verifier? How sensitive is final performance to verifier label noise?

  2. What is the inference‑side latency overhead compared against non‑reasoning embedding baselines?

  3. Your method only generates reasoning for queries while candidates are embedded without reasoning. Have you explored generating reasoning also for candidates during training / inference?

1 comment

Thank you for your careful reading and constructive feedback. We appreciate your helpful suggestions and address your concerns below.

W1/Q1: Verifier capacity and label noise.

We compared 2B, 8B, and 32B verifiers and inverted all judgments from the 32B verifier to simulate worst case label noise:

VerifierPerturbationImageVideoVisDocOverall
2BNone71.2142.3074.4965.54
8BNone71.2242.4674.4365.57
32BNone71.2942.7274.7865.77
32BFull inversion71.1042.5473.7165.31

Performance is similar across verifier sizes: 2B and 8B reach 65.54 and 65.57, versus 65.77 for 32B. The 32B verifier achieves the highest Overall score of 65.77, improving over the 2B and 8B variants by 0.23 and 0.20 points, respectively.

With the same 32B verifier, the noise perturbation reduces Overall by 0.46 points, with declines across all three modalities. Thus, verifier feedback quality matters: incorrect candidate-matching feedback weakens retrieval performance even at fixed model capacity.


W2/Q2: Inference latency.

We measured query encoding latency on the same CIRR test set using one L20X GPU, BF16, and batch size 1, retaining each method's original generation settings. Latency and length are reported as mean ± SD.

MethodCIRR Hit@1 (%)Latency (ms)Length (tokens)
VLM2Vec-2B46.850.6 ± 5.1/
UME-R1-2B53.75,165.4 ± 4,236.7263.7 ± 214.8
OPR-Embed-2B61.52,541.2 ± 1,546.5100.3 ± 63.8

Relative to VLM2Vec, OPR-Embed has 50.3× the query encoding latency and improves Hit@1 by 14.7 points. Relative to UME-R1, which also uses reasoning, it improves Hit@1 by 7.8 points while reducing mean latency by 50.8%.


W3: Trajectory mismatch and objective conflict.

Our central finding is that trajectories used in joint training with offline CoT differ from those the model generates at inference, affecting retrieval. Identifying and mitigating this mismatch does not require exhaustively ruling out every possible optimization factor in joint training.

In Figure 1 / Table 3, Oracle-CoT and Self-CoT training share initialization, training samples, joint objectives, and optimization settings; only the training CoT source differs. They achieve 54.0 and 61.3 Hit@1 with self-generated reasoning. Using language modeling and contrastive objectives together therefore does not, by itself, explain this difference. Moreover, on the same Oracle-SFT checkpoint, replacing self-generated trajectories with target-assisted Oracle trajectories raises Hit@1 from 54.0 to 78.1, without updating parameters or changing the encoder or readout. The model can extract effective representations from supplied reasoning, but its own reasoning does not realize this capability.

Appendix B.3 provides complementary distributional evidence: Self-CoT is closer to the initial student distribution; after joint training, the train-to-self NLL gaps are 0.58 and 0.08 nats for Oracle-CoT and Self-CoT training, respectively. Together, these results support a mismatch between training and inference trajectories.

We therefore apply language supervision and representation learning to the same current student trajectories. In Table 3, Decoupled SFT, which moves embedding training onto student-generated trajectories, reaches 65.1; Attentive OPSD further shares current student trajectories and reaches 67.1, without a prebuilt CoT corpus. Objective conflict and trajectory mismatch can coexist; our diagnosis and method address the directly observed training–inference trajectory gap.


Q3: Candidate-side reasoning.

We use query-side reasoning to interpret what should be retrieved, for example, which image attributes to change or preserve. A candidate alone need not pose a problem to solve; for a category name or short answer, an added rationale may offer little task-relevant information.

Our teacher sees the query and its paired target, so it can guide the student toward details needed to identify that target. Candidate-side generation needs its own objective and supervision signal, rather than a direct copy of this query-side setup. Our design uses reasoning to interpret the query's retrieval requirements, whereas RIME defines generation as retrieval-friendly rewriting.

With the final 2B checkpoint, query embeddings and candidate pool fixed, we enabled candidate reasoning only at inference on all 78 datasets:

ReasoningImage (36)Video (18)VisDoc (24)Overall (78)Candidate tokens (mean ± SD)Offline encoding cost
Query only71.2942.7274.7865.77N/A1.00×
Query + candidate69.8343.5474.0165.0563.9 ± 48.39.06×

Overall falls from 65.77 to 65.05 at 9.06× offline encoding cost, supporting query-only reasoning. Video gains 0.82 points, possibly by making temporal details explicit; in images and documents, generated text may obscure query-relevant information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.