acceptodds
Under review as a conference paper at ICLR 2027

VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

15 views

Abstract

Universal multimodal retrieval requires compact embeddings that preserve task-relevant semantic information across diverse modalities. Prior works have incorporated latent reasoning into multimodal embedding learning to refine this information before embedding extraction. However, most existing approaches remain confined to deterministic latent paths, without exploring alternative trajectories to discover better embeddings. Thus, we propose VaME (Variational Multimodal Embeddings), a framework that models latent reasoning as a learnable distribution over trajectories. Specifically, we first introduce Variational Latent Reasoning (VLR) to enable autoregressive exploration in latent space, guided by answer reconstruction through a lightweight decoder. Meanwhile, we augment the original embedding-token readout with a latent-fused embedding to facilitate exploration during subsequent reinforcement learning. Finally, we optimize latent reasoning over stochastic variational trajectories through reinforcement learning, using Semantic Decoding Reward (SDR) to favor semantically meaningful trajectories with interpretable decoded outcomes. On the 78-task MMEB-V2 benchmark, spanning image, video, and visual-document retrieval, VaME outperforms most explicit CoT-based models and all latent-reasoning baselines. VaME also demonstrates robust performance on reasoning-intensive benchmarks such as MRMR, with substantial gains after reinforcement learning. Importantly, VaME achieves these gains with at least a 4.25× inference speedup over the deterministic latent autoregressive baselines. The code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal Multimodal Retrieval, Latent Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iqAgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VaME, which replaces deterministic latent recurrence in multimodal embedding models with conditional Gaussian transitions, enabling exploration over different latent trajectories during training. The model first undergoes latent supervised fine-tuning using contrastive retrieval objectives, text reconstruction, and regularization, followed by reinforcement learning with retrieval rewards and a Semantic Decoding Reward (SDR). At inference, it follows a deterministic conditional-mean trajectory and produces a single retrieval embedding. VaME achieves competitive results on MMEB-V2 and MRMR and reports substantial throughput improvements over prior autoregressive latent-reasoning methods.

Critical strengths
  1. The core idea of VLR is appealing. Modeling latent transitions as learnable distributions provides a direct way to generate and compare alternative computational trajectories for the same input. Exploring during training while following a mean trajectory at inference is also a reasonable approach to benefiting from exploration without requiring multi-trajectory search at deployment.

  2. The training objectives form a coherent design. Beyond retrieval feedback, the framework uses text reconstruction and SDR to constrain the semantic content of latent trajectories. This provides richer feedback than retrieval rewards alone and attempts to balance discriminative representation learning with semantic preservation.

  3. The overall retrieval results are competitive. VaME-2B and VaME-7B achieve MMEB-V2 scores of 65.7 and 69.4, respectively, while VaME-7B reaches 51.4 on MRMR, outperforming the corresponding models reported in the tables. These results suggest that the framework merits further investigation.

Critical weaknesses
  1. The construction of text reconstruction targets and SDR reference answers is insufficiently clear.

    The paper states that existing answers, descriptions, or explanations are reused, but does not specify the reconstruction targets for the query and target sides of each task. For example, could both sides reconstruct the answer text in VQA, or the paired caption in caption retrieval? This raises a natural concern: if both sides frequently share the same reconstruction target, could this encourage their latent representations to converge and provide a shortcut for contrastive alignment through shared text reconstruction, rather than through the proposed trajectory exploration? The current description makes this alternative explanation difficult to assess. The construction of SDR reference answers for non-QA retrieval tasks also requires clarification.

  2. The ablations do not sufficiently establish which supervision signals make VLR effective.

    In Table 3, introducing VLR alone reduces performance from 63.8 to 63.1. With latent SFT, the deterministic model and VLR achieve 64.6 and 64.9, respectively. I would therefore like to understand which latent SFT losses enable stochastic exploration to provide additional value.

    Although Table 4 removes individual auxiliary losses, its full-model score of 65.7 corresponds to the RL-trained model, rather than the SFT-only score of 64.9. The authors should clarify whether each variant undergoes RL and provide key loss ablations before RL, separating their direct effects on SFT representations from their downstream effects on RL. Targeted comparisons with the deterministic model, where the losses are applicable, would also help identify interactions between supervision and stochastic exploration. In particular, if the w/o L_{ans} variant still uses SDR during RL, how is its latent decoder trained?

  3. The substantial inference speedup is not adequately explained.

    Both VaME and PLUME use autoregressive latent computation with KV caching, yet Figure 3 reports 23.8 versus 3.3 samples/s at the same budget of K = 8, a roughly 7.2× difference. Following the mean trajectory avoids sampling but does not eliminate the sequential backbone computations, so this design alone does not explain the magnitude of the speedup.

    Beyond the reported single-H800, batch-size-1 setting, please clarify whether model size, input/token budgets, precision, and inference implementations are matched. A brief latency breakdown would help distinguish architectural gains from implementation or measurement differences.

  4. The interpretability analysis of latent trajectories remains preliminary.

    Figure 4 presents interesting differences in decoded outputs, but does not clearly show how latent representations evolve throughout the autoregressive process. A more process-oriented visualization, such as decoded top-k tokens or semantic changes at successive latent steps, would help readers understand how the trajectories develop and relate to retrieval outcomes.

    The authors could draw on the analyses in [1–3] to strengthen the interpretation of the latent computation.

    [1] Shen et al. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP 2025.

    [2] Dilgren and Wiegreffe. Are Latent Reasoning Models Easily Interpretable? arXiv:2604.04902

    [3] Li et al. Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure. ICML 2026.

Questions for authors

I would appreciate clarification of the details raised above, particularly how text supervision targets are constructed across tasks, which losses and RL stages are included in each configuration in Tables 3–4, and what accounts for the substantial inference speedup. These details directly affect the interpretation of the method's contributions and experimental results, and the presentation could be clearer.

I also encourage further analysis of the latent trajectories. A more direct view of how they evolve during autoregressive computation would strengthen the paper. I find the VLR idea appealing and the reported results promising. If the authors' response adequately addresses these concerns, I would be willing to raise my score.

5 comments
AuthorsOpenReview ↗

(1) Response to Reviewer iqAg W1/W2 — Reconstruction Targets, Shared-Text Alignment, and the Role of SFT Supervision in VLR

We thank Reviewer iqAg for the constructive feedback.

W1 — Reconstruction Targets, SDR References, and Shared-Text Alignment

Each side uses its own final answer field from the Embed-RL[1] training annotations, containing an answer, description, or explanation. Five abbreviated examples follow:

TaskQuery inputTarget inputQuery reconstruction targetTarget reconstruction target
VQAChart + “What percentage of UK SMEs used video conferencing in 2017?”“63.3%”“63.3% of UK SMEs used video conferencing in 2017.”“The text represents the percentage 63.3%.”
Image→text retrievalImage of two white birds flying over waterPaired caption“A wide-angle photo of two white birds flying over water.”“The text describes a wide-angle scene of two white birds flying above water.”
Text→image retrievalCaption describing giraffes in an outdoor enclosure on a sunny dayPaired giraffe image“Giraffes in an outdoor enclosure on a sunny day.”“Several giraffes are in an outdoor enclosure, with one sitting in the foreground.”
Image classificationShoe-shop image + classification instruction“shoe shop, shoe-shop, shoe store”“The image shows a shoe shop.”“The text lists synonyms for a retail establishment selling shoes.”
GroundingFull image + “Select the boat region.”Cropped boat image“The boat’s bounding box is [268, 450, 703, 716].”“A boat with a blue metal frame, a rusty hull, and a lifebuoy.”

Across 1,471,320 SFT examples, only 0.0103% of pairs have identical reconstruction targets. Different texts can still share semantics: this intentionally promotes positive-pair alignment, consistent with contrastive learning. The question is what trajectory exploration adds beyond this supervision. Table 3, supplemented with MRMR and SDR-only RL results, shows:

SettingVLRLatent SFTLatent RLMMEB-V2MRMR
Deterministic AR baseline (LNCEe\mathcal{L}_{\mathrm{NCE}}^{e} only)×××63.837.0
Deterministic AR + Latent SFT (adding only Lans\mathcal{L}_{ans})×✓×64.637.9
VLR + Latent SFT (all auxiliary losses)✓✓×64.938.3
VLR + Latent SFT + Latent RL (SDR only)✓✓✓65.3 (+0.4)40.7 (+2.4)

Reconstruction improves deterministic AR by 0.8 points on MMEB-V2 and 0.9 on MRMR, supporting its alignment role and the reviewer’s explanation. SDR-only RL further improves VLR + Latent SFT by 0.4 and 2.4 points, respectively. The larger gain on reasoning-intensive retrieval supports our motivation: semantic decoding feedback guides exploration toward paths that better preserve task-relevant semantics and improve retrieval.

W2 — Which SFT Signals Make VLR Effective?

The supervision signals serve complementary roles:

SFT signalRole
LNCEe\mathcal{L}_{\mathrm{NCE}}^{e}Aligns final retrieval embeddings
LNCEz\mathcal{L}_{\mathrm{NCE}}^{z}Aligns trajectory-fused representations
LNCEZ\mathcal{L}_{\mathrm{NCE}}^{Z}Supervises retrieval at each latent step
Lans\mathcal{L}_{\mathrm{ans}}Preserves task-relevant answer and description semantics
LKL\mathcal{L}_{\mathrm{KL}}Regularizes the sampling distribution
Ldiv\mathcal{L}_{\mathrm{div}}Discourages redundancy across latent steps

In Table 3, VLR with only LNCEe\mathcal{L}_{\mathrm{NCE}}^{e} and no direct trajectory supervision reduces performance from 63.8 to 63.1. Latent SFT raises it to 64.9, supporting the need for supervision to make exploration useful. Latent SFT also improves deterministic AR from 63.8 to 64.6.

Table 4 incorrectly uses the post-RL score of 65.7 as its reference, although all loss-ablation variants were evaluated before RL. We will correct the SFT-only reference to 64.9:

#ModelImageVideoVisDocAll
1VaME-2B (w/o RL)69.045.573.364.9
2w/o LNCEz\mathcal{L}_{\mathrm{NCE}}^{z}68.845.173.664.8
3w/o LNCEZ\mathcal{L}_{\mathrm{NCE}}^{Z}69.346.273.665.3
4w/o Lans\mathcal{L}_{\mathrm{ans}}68.845.773.164.8
5w/o LKL\mathcal{L}_{\mathrm{KL}}68.445.272.764.4
6w/o Ldiv\mathcal{L}_{\mathrm{div}}68.745.573.164.7

Removing LNCEZ\mathcal{L}_{\mathrm{NCE}}^{Z} raises the SFT score to 65.3, but subsequent RL under the same settings reaches only 65.1 versus the full model’s 65.7. This suggests that its main benefit here is supporting subsequent trajectory optimization rather than immediate SFT performance.

The w/o Lans\mathcal{L}_{\mathrm{ans}} variant undergoes only SFT, with no RL or SDR. Thus, no decoder lacking reconstruction training is used to compute SDR.

[1] Jiang et al. Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings. ECCV 2026.

AuthorsOpenReview ↗

(2) Response to Reviewer iqAg W3 — Sources of Inference Speedup, Engineering Optimizations, and Fair Comparison Settings

W3 — Sources of Inference Speedup and Comparison Settings

Thank you for raising this concern. Both VaME and PLUME perform K=8 autoregressive latent steps, so mean-trajectory inference alone cannot explain the reported speedup. Below, we clarify implementation gains, comparison settings, and the timing protocol.

The throughput comparison includes engineering optimizations. We distinguish local measurements from published results:

Method (K=8)Implementation / SourceThroughput (samples/s)
VaMEBase implementation5.9
VaMEKernel fusion9.3
VaMEKernel fusion + CUDA Graphs23.8
PLUMEOriginal implementation, locally re-evaluated3.0
PLUMECUDA Graphs, same environment7.2
PLUMESame kernel fusion + CUDA Graphs, same environment9.3
PLUMEReported in TTE-Flash3.3
TTE-FlashReported in the original paper5.6

For VaME, kernel fusion increases throughput from 5.9 to 9.3 samples/s, and CUDA Graphs further raises it to 23.8—a 4.03× improvement over the base implementation. PLUME also improves from 3.0 to 7.2 with CUDA Graphs and to 9.3 with kernel fusion added. Thus, engineering optimizations substantially benefit both methods. With both optimizations enabled, VaME achieves 2.56× PLUME’s throughput. We will revise the speedup claim and document the comparison details accordingly.

The local comparison uses the following settings, rather than assuming equal compute budgets solely from the same K:

SettingVaMEPLUME
Model size2B2B
BackboneQwen2-VL-2BQwen2-VL-2B
PrecisionBF16BF16
Batch size11
Latent steps K88
KV cacheEnabledEnabled
AttentionFlashAttention 2FlashAttention 2
Image/document min_pixels3,1363,136
Image/document max_pixels802,816802,816
Video per-frame min_pixels3,1363,136
Video per-frame max_pixels286,720286,720
Video frames88
Inference pathDeterministic mean trajectoryOfficial deterministic latent inference
OptimizationsBase, fusion, and fusion + CUDA Graphs reported separatelyOriginal, fusion, and fusion + CUDA Graphs reported separately

Timing uses 3,000 real examples randomly sampled by modality from MMEB-V2 and MRMR. Each model is measured over three repeated runs to reduce measurement variability.

AuthorsOpenReview ↗

(3) Response to Reviewer iqAg W4 — Stepwise Latent Trajectories and Retrieval Outcomes

Thank you for the suggestion. Figure 4 mainly compares decoded outputs of complete trajectories, without showing their evolution during autoregressive computation. We therefore add a step-wise analysis using VaME-2B.

Setup. We randomly select five test examples from MMEB-V2 with seed 0. At step t, the hidden-state prefix through that step is passed to the trained, frozen decoder for greedy decoding. The decoder receives only continuous representations and a fixed decoding prompt. Three cases are shown below. Rankings are reported for the embedding-token representation (e) and trajectory-fused representation (z).

Case 1: Comparing magnetic forces — ScienceQA, sample 9

Question: Which statement correctly compares the magnetic forces between the two pairs of magnets?
Reference answer: The magnetic force is stronger in the first pair.

StepDecoded outputCorrect-answer rank (e/z)
1Two magnets are in a magnetic field, with the left magnet most prominent.4 / 3
2The ratio of the magnets is 1:1.2 / 2
3The magnetic force between the magnets in the first pair is 1.2 N.1 / 2
4The magnetic force is stronger in the first pair than in the second.1 / 2
7The magnetic force is stronger in the first pair than in the second.1 / 1
8The magnetic force is stronger in the first pair than in the second.1 / 2

Decoded content progresses from object descriptions and an inaccurate numerical claim to a task-relevant comparison. The correct-answer rank improves from 4 to 1 for e, while the rank for z fluctuates.

Case 2: Grounding the brightly lit curtain region — Visual7W-Pointing, sample 554

Question: Which part of the curtain is illuminated by bright light?
Correct target: The annotated crop of the brightly lit curtain region, retrieved from the full pool of 1,000 region images.

StepDecoded outputCorrect-region rank (e/z)
1A bright window with partially open curtains.2 / 2
2Bright, soft light passes through the window.2 / 2
4The brightly lit region is the upper part of the window.1 / 1
7The relevant part is at the top of the image, from the top of the window to the top of the curtain.1 / 1
8Locate the region labeled “window” and output coordinates [100,100,900,900].1 / 1

Decoding shifts from a general window description to a specific region. Both representations improve the correct crop’s rank from 2 to 1 at step 4.

Case 3: Recognizing a Basset Hound — ImageNet-1K, sample 276

Correct class: Basset Hound, retrieved from all 1,000 classes.

StepDecoded outputCorrect-class rank (e/z)
1A Golden Retriever and a Beagle stand on the grass.2 / 1
3A Golden Retriever and a Sussex Spaniel stand outdoors.2 / 1
4A Golden Retriever and a Basset Hound stand outdoors.1 / 1
8The image is classified as containing a Basset Hound.1 / 1

Decoding shifts from incorrect breeds to the correct breed. When “Basset Hound” appears at step 4, the correct-class rank for e also improves from 2 to 1.

Case 4: Identifying the first celestial body shown — Video-MME, sample 136

Question: Which celestial body is shown first in the video?
Correct answer: Earth, among four candidates.

StepDecoded outputCorrect-answer rank (e/z)
1Earth viewed from space, in the foreground.2 / 2
2An animation of Earth focusing on its surface and atmosphere.2 / 1
5Earth viewed from space, with a largely unchanged scene.1 / 1
6The first celestial body shown in the video is Earth.1 / 1
8The first celestial body shown in the video is Earth.1 / 1

The trajectory-fused representation (z) reaches rank 1 at step 2, and the embedding-token representation (e) at step 5. Later decoded outputs directly answer the question.

Case 5: Retrieving a document containing an order number — ViDoRe_docvqa, sample 19

Question: What is the order number?
Relevant document ID: 19, retrieved from the full collection of 500 document pages.

StepDecoded outputRelevant-document rank (e/z)
1A stack of blue and white shipping containers.3 / 7
2Packaging boxes with barcodes and product labels.2 / 6
5A product order form with an order-number field labeled “No.”1 / 3
6A black office chair with a red seat cushion.1 / 2
7A document or form with an “order no.” field.1 / 3
8A document or form containing “Order No. 40549”.1 / 2
Reviewer_YfzYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers latent reasoning for universal multimodal embeddings, where an MLLM runs a few hidden "thinking" steps before producing one retrieval vector. The key idea is that these hidden steps should be sampled from a learnable Gaussian distribution rather than follow one fixed path, so the model can explore alternative trajectories during training. The pipeline first aligns trajectories through retrieval and answer-reconstruction SFT. It then refines them with GRPO, using a retrieval reward and an LLM-judged decoding reward. At inference only the mean trajectory is used, and the paper reports top averages on MMEB-V2 and MRMR.

Critical strengths
  1. Modeling each latent step as a conditional Gaussian makes continuous latent reasoning a proper policy, so GRPO-style RL applies directly in hidden space.
  2. Stochasticity is confined to training, so deployment keeps a cheap single-vector interface with one mean-path pass and no text generation.
  3. The evaluation is broad: the 78-task MMEB-V2 at two model scales plus the reasoning-intensive MRMR, with ablations on components, objectives, and rewards.
Critical weaknesses
  1. Variational Sampling Trajectory alone is not the main point, it must be combined with SFT and RL to have real effect. But there are missing variant baseline Deterministic AR + SFT + RL or Fixed-noise sampling + SFT + RL. What happens if these two new baselines is comparable with the proposed method?

  2. The ≥4.25× speedup is not explained by the method. At inference VaME runs a deterministic 8-step recurrence, structurally identical to the AR baselines, yet it is ~7× faster than PLUME at the same K, which points to implementation differences. So a real K = 8 deterministic comparison is unclear.

  3. There are too many hyperparamter settings (about 13), and there is no sentivity and robustness analysis of the methods with the hyperparameters, at least with some new key ones.

Questions for authors

As in weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer YfzY — Deterministic and Fixed-Noise Baselines, Inference Speedup, and Hyperparameter Sensitivity in VaME

Thank you to Reviewer YfzY for the careful review and valuable suggestions. We respond to the three concerns one by one below. W1 — Would Deterministic AR or Fixed-Noise Sampling Match VaME under SFT and RL?

Thank you for suggesting these two comparisons, which help clarify where VaME's performance gains come from.

Deterministic AR. For the same query, strictly deterministic AR generates the same trajectory on every run. Under identical reward conditions, all trajectories receive the same score, so their within-group advantages are zero. Therefore, starting from the same SFT checkpoint, Latent GRPO cannot provide an additional reward-learning signal. The corresponding baseline is the deterministic AR model with Latent SFT only. We restate the relevant results from Table 3 below.

MethodMMEB-V2MRMR
Deterministic AR + Latent SFT64.637.9
VLR + Latent SFT64.938.3
VaME (VLR + Latent SFT + Latent RL)65.740.5

Fixed-noise sampling. After fixing the standard deviation of the Gaussian distribution, each rollout can still sample noise independently and produce a different trajectory. GRPO can therefore be applied directly for a comparable ablation. We report the new matched ablation below and will include it in the final manuscript. σ\sigma denotes the standard deviation of the Gaussian noise.

MethodNoise standard deviationMMEB-V2MRMR
Fixed Gaussian noise0.25σ0.25\sigma64.737.9
Fixed Gaussian noise0.5σ0.5\sigma64.638.3
Fixed Gaussian noiseσ\sigma64.938.5
Fixed Gaussian noise2σ2\sigma65.338.1
Fixed Gaussian noise4σ4\sigma64.637.9
VaMEOurs (learnable)65.740.5

Under the same SFT and RL settings, VaME outperforms all five fixed-noise settings that we tested on both MMEB-V2 and MRMR. This result indicates that, beyond introducing stochasticity for latent-trajectory exploration, learning the noise magnitude from each input promotes meaningful exploration more effectively than random search.

W2 — Explaining VaME's Speedup over PLUME

Thank you for pointing this out. The reported speedup includes infrastructure-level optimizations. Both VaME and PLUME use K=8 latent recurrence steps at inference. Beyond our base AR implementation, we apply kernel fusion to reduce GPU kernel launches and CUDA Graphs to reduce scheduling overhead, bucketing and padding variable-length inputs for graph replay. These optimizations were absent from the original AR baseline.

The table below reports VaME’s throughput before and after optimization and separates re-evaluated baselines from published results. We will document the optimizations, baseline configurations, and measurement protocol in the appendix, clarifying that the speedup reflects both architectural and implementation differences.

Method(K=8K=8)ImplementationThroughput (samples/s)
VaMEBase implementation5.9
VaMEKernel fusion9.3
VaMEKernel fusion + CUDA Graphs23.8
PLUMERe-evaluated original implementation3.0
PLUMECUDA Graphs in the same environment7.2
PLUMEReported in the TTE-Flash paper3.3
TTE-FlashReported in the TTE-Flash paper5.6

W3 — Sensitivity to Hyperparameters

Thank you for pointing out that the multiple loss and reward terms may increase the hyperparameter-tuning burden. Table 9 also reports standard training settings such as the learning rate and batch size. Specifically, Latent SFT contains five loss weights, whose default values are λz=1\lambda_z=1, λZ=1\lambda_Z=1, λans=1\lambda_{\mathrm{ans}}=1, λKL=0.1\lambda_{\mathrm{KL}}=0.1, and λdiv=0.1\lambda_{\mathrm{div}}=0.1. The SDR reward weight in Latent RL is set to λSDR=0.2\lambda_{\mathrm{SDR}}=0.2. We additionally conduct controlled one-variable-at-a-time sensitivity experiments, as shown below.

HyperparameterTested valueMMEB-V2
λz\lambda_z0.565.2
1.0 (default)65.7
2.065.8
λZ\lambda_Z0.565.4
1.0 (default)65.7
2.065.2
λans\lambda_{\mathrm{ans}}0.565.4
1.0 (default)65.7
2.066.1
λKL\lambda_{\mathrm{KL}}0.0565.4
0.1 (default)65.7
0.264.8
λdiv\lambda_{\mathrm{div}}0.0564.9
0.1 (default)65.7
0.265.3
λSDR\lambda_{\mathrm{SDR}}0.0565.8
0.165.3
0.2 (default)65.7
Learning rate5×10−75\times10^{-7}65.1
1×10−61\times10^{-6} (default)65.7
2×10−62\times10^{-6}65.3

Overall, VaME remains stable across the tested settings, demonstrating low hyperparameter sensitivity without requiring precise hyperparameter tuning.

Thank the authors for their effort. Some of my concerns are partially addressed. There are some remaining points:

  • The MMEB-V2 (the measure appears in the two above tables) improvement is only 0.4 points over the strongest fixed-noise baseline (65.3 vs 65.7), and with the ablation, many configurations have performance smaller than or equal to the baseline (<=65.3), and the method has some fluctuation > 0.4. This suggests some disadvantage in the applicability of the method, when carefully designed baselines can be comparable to a learned method with heavy grid search for a lot of hyperparams. Might be in different datasets, default hyperparams might be worser than a baseline.
  • The speeding up is with both coding optimization and might be with the method, so the original claim for >= 4.25 speeding up is not convincing anymore. Also, why does another method have no Kernel fusion optimization?
AuthorsOpenReview ↗

Response to Reviewer YfzY

Thank you for the further feedback. We acknowledge that the MMEB-V2 gain over the strongest fixed-noise baseline is modest. One possible explanation is that MMEB-V2 covers broad semantic matching and perception tasks rather than specifically targeting reasoning. Its original paper selects video QA tasks with relatively low dependence on knowledge or reasoning [1]. In contrast, MRMR targets expert knowledge, theorem linking, and contradiction identification [2].

Against the strongest fixed-noise result on each benchmark, VaME improves MMEB-V2 by 0.4 points (65.3→65.7) and MRMR by 2.0 points (38.5→40.5). This is consistent with trajectory exploration benefiting reasoning-intensive tasks more. We additionally evaluated the same ablation checkpoints on MR²-Bench [3], RMIR [4], and COCO-CN [5], without changing hyperparameters. VaME exceeds their best fixed-noise results by 1.0, 1.2, and 0.4 points, respectively. Gains therefore extend beyond MMEB-V2 but vary by task, with smaller improvements on benchmarks emphasizing semantic matching and perception, as expected.

MethodNoise standard deviationMMEB-V2MRMRMR²-Bench (nDCG@10)RMIR (R@20)COCO-CN (i2t/t2i Avg)
Fixed Gaussian noise0.25σ0.25\sigma64.737.923.740.477.3
Fixed Gaussian noise0.5σ0.5\sigma64.638.323.740.677.3
Fixed Gaussian noiseσ\sigma64.938.523.740.477.2
Fixed Gaussian noise2σ2\sigma65.338.123.940.577.4
Fixed Gaussian noise4σ4\sigma64.637.923.540.077.0
VaMEOurs (learnable)65.740.524.941.877.8

Regarding hyperparameter sensitivity and tuning cost, some configurations indeed underperform the fixed-noise baseline. For example, λKL=0.2\lambda_{\mathrm{KL}}=0.2 doubles the default and may overly constrain the sampling distribution; λdiv=0.05\lambda_{\mathrm{div}}=0.05 halves the default and may weaken trajectory diversity; and a learning rate of 5×10−75\times10^{-7} halves the default and may yield insufficient optimization within the same budget. These are one-variable-at-a-time sensitivity experiments around the default, not a joint grid search. Default weights use common scales such as 1 and 0.1 and were not selected through large-scale joint search. Some configurations reach 66.1, above the default 65.7, yet we report the default. We will distinguish its performance advantage from robustness across settings, without implying that every configuration retains this advantage.

We also agree that the reported speedup includes engineering contributions and cannot be attributed entirely to the method. As detailed in our response to Reviewer iqAg, after matching model size, backbone, precision, batch size, latent steps, and input budgets, we enabled kernel fusion and CUDA Graphs for PLUME, raising throughput from 3.0 to 9.3 samples/s. PLUME also benefits from these optimizations; we have now added this previously missing configuration.

For VaME, kernel fusion raises throughput from 5.9 to 9.3 samples/s, and CUDA Graphs further increases it to 23.8 samples/s, a 4.03× improvement over the base implementation. We will revise the “≥4.25×” claim and report implementation configurations separately. With kernel fusion and CUDA Graphs enabled for both methods, VaME achieves 23.8 samples/s versus PLUME’s 9.3 samples/s, yielding approximately 2.56× speedup.

Method (K=8)ImplementationThroughput (samples/s)
VaMEBase implementation5.9
VaMEKernel fusion9.3
VaMEKernel fusion + CUDA Graphs23.8
PLUMEOriginal implementation, locally re-evaluated3.0
PLUMECUDA Graphs7.2
PLUMEKernel fusion + CUDA Graphs9.3

Both use Qwen2-VL-2B, BF16, batch size=1, K=8, KV caching, and FlashAttention 2, with matched image and video input budgets. Timing uses 3,000 real examples randomly sampled by modality from MMEB-V2 and MRMR, with three runs per model.

References

[1] Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. TMLR 2026.

[2] Zhang et al. MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval. ICLR 2026.

[3] Zhou et al. MR²-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval. EMNLP 2026.

[4] Li et al. RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval. CVPR 2026.

[5] Li et al. COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval. IEEE Transactions on Multimedia, 2019.

Reviewer_c3vxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VaME (Variational Multimodal Embeddings), a two‑stage framework for universal multimodal retrieval embeddings. Unlike existing latent‑reasoning embedding models which use fixed deterministic reasoning trajectories, VaME models latent reasoning as a learnable conditional Gaussian distribution over autoregressive latent trajectories.

Critical strengths
  1. Prior latent‑thinking embedding models follow one fixed deterministic computation path. This work introduces conditional Gaussian stochastic latent transitions to sample multiple reasoning trajectories during training

  2. The proposed method enables latent‑space exploration without explicit text generation and preserves single‑vector retrieval interface, which is a neat conceptual advance for implicit reasoning embeddings.

  3. Pure retrieval reward only measures embedding similarity and cannot guarantee the latent reasoning carries correct semantic content. SDR leverages a lightweight frozen decoder to decode latent states and evaluate answer correctness.

Critical weaknesses
  1. Latent RL requires sampling G=8 trajectories for each query during training, raising training compute cost compared to deterministic latent‑reasoning baselines.

  2. The paper focuses on inference‑time speedup but does not systematically analyse or discuss training‑time computational overhead, memory footprint, or trade‑offs between G and performance.

  3. The model samples stochastic trajectories for training but uses mean‑mode deterministic trajectories for test. Although empirical results look good, there is limited analysis for distribution mismatch between training‑time sampled trajectories and inference‑time mean trajectory; there is no diagnostic experiment quantifying performance gap between sampled rollouts vs mean trajectory outputs.

Questions for authors
  1. During Latent RL you sample G=8 trajectories for each query. Could you elaborate on training‑time compute / memory overhead compared to deterministic latent reasoning baselines? Have you ablated the number of trajectories G, what is the performance‑compute trade‑off?

  2. Have you measured retrieval performance gap between sampled trajectories and mean‑trajectory outputs? Are there cases where mean‑mode degrades quality compared to sampled rollouts?

  3. In ablation, adding SDR brings large MRMR improvement but slight drop when combined with retrieval reward on MRMR. Could you give an explanation for this slight performance drop when combining both rewards?

1 comment
AuthorsOpenReview ↗

Response to Reviewer c3vx — Training Cost, Trajectory Sampling, and Reward Ablations in VaME

We thank Reviewer c3vx for carefully reading our paper and for the thoughtful feedback. We address your questions and concerns below. All experiments in this response use Qwen2-VL-2B for efficient ablations.

W1, W2, Q1 — Training-Time Compute and Memory Overhead

Latent RL adds training cost compared with the deterministic latent baseline, but this cost is necessary to move beyond what SFT alone can achieve. The deterministic baseline lacks the intrinsic randomness needed to explore diverse latent paths. VaME enables this exploration through variational trajectories. Just as LLMs use preference alignment during post-training to better match human intent, VaME uses retrieval and semantic decoding rewards to adapt to more complex downstream retrieval tasks.

We report total training GPU-hours and peak memory per GPU under matched settings, using Qwen2-VL-2B with a batch size of 512 for SFT and 64 for RL.

MethodMMEB-V2MRMRTotal H100 GPU-hoursPeak memory per GPU
Deterministic latent baseline64.637.964 (8 × 8 h)≈65 GB
VaME (Latent SFT + Latent RL)65.740.5108 (8 × 9 h for SFT + 8 × 4.5 h for RL)≈65 GB for SFT, ≈35 GB for RL

W2, Q1 — Performance–Compute Trade-off with Different GG

In Latent RL, GG is the number of trajectories sampled per query. Starting from the same Latent SFT checkpoint, we compare retrieval performance, RL training cost, and peak memory per GPU for G∈{2,4,8,16}G \in \{2, 4, 8, 16\}, holding the number of queries per step and the number of RL updates fixed. Increasing GG from 8 to 16 raised RL cost by about 36% (from 36 to 49 H100 GPU-hours), left MMEB-V2 unchanged, and improved MRMR by only 0.2 points.

This pattern is consistent with GRPO: larger groups help stabilize estimates of the within-group reward mean and variance, but the gains in retrieval performance diminish as GG grows. We therefore use G=8G=8 by default, consistent with the GRPO group size used in CoLaR [1], UME-R1 [2], and RIME [3].

Trajectories per query (GG)MMEB-V2MRMRRL cost (H100 GPU-hours)Peak memory per GPU (GB)
264.537.2≈20.0≈23
465.039.4≈25.0≈29
8 (default)65.740.5≈36.0≈35
1665.740.7≈49.0≈45

W3, Q2 — Sampled versus Mean-Trajectory Inference

VaME samples latent trajectories during training. At inference, it takes the conditional mean at each step to obtain a deterministic trajectory. We compare the two modes using the same checkpoint, the same number of inference steps, and one trajectory per input. For sampled inference, we report the mean and standard deviation (SD) across 16 random seeds.

Inference modeMMEB-V2MRMR
Mean trajectory (default)65.740.5
Sampled trajectory (mean ± SD)65.8 ± 0.2340.8 ± 0.74
Difference (sampled average − mean trajectory)+0.1+0.3

The mean trajectory scored 0.1 points lower on MMEB-V2 and 0.3 points lower on MRMR than sampled rollouts on average. We use it by default because it gives each input a consistent embedding across runs, which is important for large-scale embedding extraction.

Q3 — Interaction between Retrieval Reward and SDR

In Table 5, adding the retrieval reward to SDR lowered MRMR from 40.7 to 40.5 (−0.2) and raised MMEB-V2 from 65.3 to 65.7 (+0.4). A single run cannot establish whether the 0.2-point drop is significant. We therefore report the mean and standard deviation across eight RL training seeds.

Reward settingMMEB-V2 (Table 5)MRMR (Table 5)MMEB-V2 (mean ± SD)MRMR (mean ± SD)
SFT baseline64.938.3——
SDR only65.340.765.2 ± 0.1640.4 ± 0.62
Retrieval reward + SDR65.740.565.6 ± 0.2040.6 ± 0.44

Across training seeds, adding the retrieval reward raised the mean score from 65.2 to 65.6 on MMEB-V2 and from 40.4 to 40.6 on MRMR. MRMR covers a narrower range of tasks than MMEB-V2, so its scores may fluctuate more. The retrieval reward may yield relatively less consistent gains than SDR on complex retrieval tasks, underscoring the importance of exploring intermediate latent trajectories.

References

[1] Tan et al. Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains. NeurIPS 2025.

[2] Lan et al. UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings. ICLR 2025.

[3] Wu et al. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings. ACMMM 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revisions in Response to Your Feedback and a Request to Reconsider the Score

Thank you for reading our paper carefully and raising specific concerns about the throughput comparison, the ablations, and the training setup. The original speedup claim went beyond what our comparison supported, and the ablation table did not clearly distinguish SFT from RL. We revised the manuscript to address these issues and incorporated the additional experiments and analyses from the rebuttal.

Blue text marks additions or revisions. Red downward triangles ▼ mark where original content was removed or replaced.

Main text

  1. Contribution and scope of the speedup claim. We withdrew the original claim of at least 4.25× speedup because it combined local measurements with published results obtained under different settings. The abstract, introduction, and conclusion now emphasize the central contribution. To our knowledge, VaME is the first to apply latent RL to universal multimodal embeddings. Adding latent RL to latent SFT improves VaME-2B by 0.8 points on MMEB-V2 and 2.2 points on MRMR.
  2. Training and deployment. The introduction now explains how conditional Gaussian sampling enables trajectory exploration, how SFT grounds those trajectories, and how latent RL refines them. At deployment, each input follows one mean trajectory and yields one retrieval embedding. Section 4 links to examples of the answer reconstruction targets.
  3. Training objective ablations. Section 5.3 and Table 4 now report SFT and RL results separately. Removing LNCEZ\mathcal{L}_{\mathrm{NCE}}^{Z} raises the SFT score, but the resulting variant scores below the full model after RL. The ablation without answer reconstruction ends at SFT because SDR requires a decoder trained to generate answers.
  4. Mean versus sampled inference. Section 6 and Table 6 compare the two modes using the same checkpoint and readout, with one eight-step trajectory per input. Across 16 seeds, sampling gains 0.1 points on MMEB-V2 and 0.3 points on MRMR on average, possibly because it better matches training. We retain mean inference by default to keep embeddings consistent across runs.
  5. Inference budget and trajectory analysis. Figure 3 now uses locally measured throughput for optimized PLUME and omits the published TTE-Flash timings. Section 6 explains the retrieval–throughput trade-off behind the eight-step default. The discussion of Figure 4 also points to the stepwise analysis in Appendix E.

Appendix

  1. A.1–A.2 Training data and supervision. We clarify deduplication, entity-level splitting, and the data used at each training stage. Ten paired examples show the query and target annotations. We also report their exact-match rate and explain which reference answers SDR uses.
  2. B.1–B.3 Training details. We clarify what each SFT loss supervises, how the reconstruction loss is normalized, and how the RL rewards and policy updates are implemented. The configuration table now gives the training set sizes, loss weights, and clipping parameters.
  3. C.1 Fixed-noise baselines. We compare learned noise with five fixed noise scales under the same SFT and RL settings. We evaluate the same checkpoints on five benchmarks, adding MR²-Bench, RMIR, and COCO-CN to MMEB-V2 and MRMR.
  4. C.2 Training cost. We report GPU-hours for VaME and the deterministic baseline, along with retrieval scores, RL cost, and peak memory for 2, 4, 8, and 16 trajectories per query. These results explain our choice of eight trajectories.
  5. C.3–C.4 Reward ablations and sensitivity. We repeat the reward ablation across 8 training seeds. The mean MRMR result reverses the ordering seen in the single run. We also report sensitivity to individual hyperparameters, including settings that match or fall below the fixed-noise baseline.
  6. D Throughput under matched conditions. We report local measurements for VaME and PLUME on one H800 with the same backbone and input budget, including the effect of kernel fusion and CUDA Graphs. We describe PLUME’s expert routing and post-readout decoding and label the TTE-Flash 8-loop result as a published measurement.
  7. E Stepwise decoding and retrieval. Five cases include the original inputs and sample identifiers, the decoder prompt, and intermediate decoded outputs alongside target ranks for both readouts.

Your detailed feedback helped us make the comparisons clearer and narrow the claims to what the evidence supports. If you feel the revision addresses your main concerns, we would be grateful if you could revisit your assessment during the discussion and consider a higher score where warranted. Thank you for taking the time to consider the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.