- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VaME, which replaces deterministic latent recurrence in multimodal embedding models with conditional Gaussian transitions, enabling exploration over different latent trajectories during training. The model first undergoes latent supervised fine-tuning using contrastive retrieval objectives, text reconstruction, and regularization, followed by reinforcement learning with retrieval rewards and a Semantic Decoding Reward (SDR). At inference, it follows a deterministic conditional-mean trajectory and produces a single retrieval embedding. VaME achieves competitive results on MMEB-V2 and MRMR and reports substantial throughput improvements over prior autoregressive latent-reasoning methods.
Critical strengths
-
The core idea of VLR is appealing. Modeling latent transitions as learnable distributions provides a direct way to generate and compare alternative computational trajectories for the same input. Exploring during training while following a mean trajectory at inference is also a reasonable approach to benefiting from exploration without requiring multi-trajectory search at deployment.
-
The training objectives form a coherent design. Beyond retrieval feedback, the framework uses text reconstruction and SDR to constrain the semantic content of latent trajectories. This provides richer feedback than retrieval rewards alone and attempts to balance discriminative representation learning with semantic preservation.
-
The overall retrieval results are competitive. VaME-2B and VaME-7B achieve MMEB-V2 scores of 65.7 and 69.4, respectively, while VaME-7B reaches 51.4 on MRMR, outperforming the corresponding models reported in the tables. These results suggest that the framework merits further investigation.
Critical weaknesses
-
The construction of text reconstruction targets and SDR reference answers is insufficiently clear.
The paper states that existing answers, descriptions, or explanations are reused, but does not specify the reconstruction targets for the query and target sides of each task. For example, could both sides reconstruct the answer text in VQA, or the paired caption in caption retrieval? This raises a natural concern: if both sides frequently share the same reconstruction target, could this encourage their latent representations to converge and provide a shortcut for contrastive alignment through shared text reconstruction, rather than through the proposed trajectory exploration? The current description makes this alternative explanation difficult to assess. The construction of SDR reference answers for non-QA retrieval tasks also requires clarification.
-
The ablations do not sufficiently establish which supervision signals make VLR effective.
In Table 3, introducing VLR alone reduces performance from 63.8 to 63.1. With latent SFT, the deterministic model and VLR achieve 64.6 and 64.9, respectively. I would therefore like to understand which latent SFT losses enable stochastic exploration to provide additional value.
Although Table 4 removes individual auxiliary losses, its full-model score of 65.7 corresponds to the RL-trained model, rather than the SFT-only score of 64.9. The authors should clarify whether each variant undergoes RL and provide key loss ablations before RL, separating their direct effects on SFT representations from their downstream effects on RL. Targeted comparisons with the deterministic model, where the losses are applicable, would also help identify interactions between supervision and stochastic exploration. In particular, if the
w/o L_{ans}variant still uses SDR during RL, how is its latent decoder trained? -
The substantial inference speedup is not adequately explained.
Both VaME and PLUME use autoregressive latent computation with KV caching, yet Figure 3 reports 23.8 versus 3.3 samples/s at the same budget of K = 8, a roughly 7.2× difference. Following the mean trajectory avoids sampling but does not eliminate the sequential backbone computations, so this design alone does not explain the magnitude of the speedup.
Beyond the reported single-H800, batch-size-1 setting, please clarify whether model size, input/token budgets, precision, and inference implementations are matched. A brief latency breakdown would help distinguish architectural gains from implementation or measurement differences.
-
The interpretability analysis of latent trajectories remains preliminary.
Figure 4 presents interesting differences in decoded outputs, but does not clearly show how latent representations evolve throughout the autoregressive process. A more process-oriented visualization, such as decoded top-k tokens or semantic changes at successive latent steps, would help readers understand how the trajectories develop and relate to retrieval outcomes.
The authors could draw on the analyses in [1–3] to strengthen the interpretation of the latent computation.
[1] Shen et al. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP 2025.
[2] Dilgren and Wiegreffe. Are Latent Reasoning Models Easily Interpretable? arXiv:2604.04902
[3] Li et al. Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure. ICML 2026.
Questions for authors
I would appreciate clarification of the details raised above, particularly how text supervision targets are constructed across tasks, which losses and RL stages are included in each configuration in Tables 3–4, and what accounts for the substantial inference speedup. These details directly affect the interpretation of the method's contributions and experimental results, and the presentation could be clearer.
I also encourage further analysis of the latent trajectories. A more direct view of how they evolve during autoregressive computation would strengthen the paper. I find the VLR idea appealing and the reported results promising. If the authors' response adequately addresses these concerns, I would be willing to raise my score.
5 comments
(2) Response to Reviewer iqAg W3 — Sources of Inference Speedup, Engineering Optimizations, and Fair Comparison Settings
W3 — Sources of Inference Speedup and Comparison Settings
Thank you for raising this concern. Both VaME and PLUME perform K=8 autoregressive latent steps, so mean-trajectory inference alone cannot explain the reported speedup. Below, we clarify implementation gains, comparison settings, and the timing protocol.
The throughput comparison includes engineering optimizations. We distinguish local measurements from published results:
| Method (K=8) | Implementation / Source | Throughput (samples/s) |
|---|---|---|
| VaME | Base implementation | 5.9 |
| VaME | Kernel fusion | 9.3 |
| VaME | Kernel fusion + CUDA Graphs | 23.8 |
| PLUME | Original implementation, locally re-evaluated | 3.0 |
| PLUME | CUDA Graphs, same environment | 7.2 |
| PLUME | Same kernel fusion + CUDA Graphs, same environment | 9.3 |
| PLUME | Reported in TTE-Flash | 3.3 |
| TTE-Flash | Reported in the original paper | 5.6 |
For VaME, kernel fusion increases throughput from 5.9 to 9.3 samples/s, and CUDA Graphs further raises it to 23.8—a 4.03× improvement over the base implementation. PLUME also improves from 3.0 to 7.2 with CUDA Graphs and to 9.3 with kernel fusion added. Thus, engineering optimizations substantially benefit both methods. With both optimizations enabled, VaME achieves 2.56× PLUME’s throughput. We will revise the speedup claim and document the comparison details accordingly.
The local comparison uses the following settings, rather than assuming equal compute budgets solely from the same K:
| Setting | VaME | PLUME |
|---|---|---|
| Model size | 2B | 2B |
| Backbone | Qwen2-VL-2B | Qwen2-VL-2B |
| Precision | BF16 | BF16 |
| Batch size | 1 | 1 |
| Latent steps K | 8 | 8 |
| KV cache | Enabled | Enabled |
| Attention | FlashAttention 2 | FlashAttention 2 |
Image/document min_pixels | 3,136 | 3,136 |
Image/document max_pixels | 802,816 | 802,816 |
Video per-frame min_pixels | 3,136 | 3,136 |
Video per-frame max_pixels | 286,720 | 286,720 |
| Video frames | 8 | 8 |
| Inference path | Deterministic mean trajectory | Official deterministic latent inference |
| Optimizations | Base, fusion, and fusion + CUDA Graphs reported separately | Original, fusion, and fusion + CUDA Graphs reported separately |
Timing uses 3,000 real examples randomly sampled by modality from MMEB-V2 and MRMR. Each model is measured over three repeated runs to reduce measurement variability.
(3) Response to Reviewer iqAg W4 — Stepwise Latent Trajectories and Retrieval Outcomes
Thank you for the suggestion. Figure 4 mainly compares decoded outputs of complete trajectories, without showing their evolution during autoregressive computation. We therefore add a step-wise analysis using VaME-2B.
Setup. We randomly select five test examples from MMEB-V2 with seed 0. At step t, the hidden-state prefix through that step is passed to the trained, frozen decoder for greedy decoding. The decoder receives only continuous representations and a fixed decoding prompt. Three cases are shown below. Rankings are reported for the embedding-token representation (e) and trajectory-fused representation (z).
Case 1: Comparing magnetic forces — ScienceQA, sample 9
Question: Which statement correctly compares the magnetic forces between the two pairs of magnets?
Reference answer: The magnetic force is stronger in the first pair.
| Step | Decoded output | Correct-answer rank (e/z) |
|---|---|---|
| 1 | Two magnets are in a magnetic field, with the left magnet most prominent. | 4 / 3 |
| 2 | The ratio of the magnets is 1:1. | 2 / 2 |
| 3 | The magnetic force between the magnets in the first pair is 1.2 N. | 1 / 2 |
| 4 | The magnetic force is stronger in the first pair than in the second. | 1 / 2 |
| 7 | The magnetic force is stronger in the first pair than in the second. | 1 / 1 |
| 8 | The magnetic force is stronger in the first pair than in the second. | 1 / 2 |
Decoded content progresses from object descriptions and an inaccurate numerical claim to a task-relevant comparison. The correct-answer rank improves from 4 to 1 for e, while the rank for z fluctuates.
Case 2: Grounding the brightly lit curtain region — Visual7W-Pointing, sample 554
Question: Which part of the curtain is illuminated by bright light?
Correct target: The annotated crop of the brightly lit curtain region, retrieved from the full pool of 1,000 region images.
| Step | Decoded output | Correct-region rank (e/z) |
|---|---|---|
| 1 | A bright window with partially open curtains. | 2 / 2 |
| 2 | Bright, soft light passes through the window. | 2 / 2 |
| 4 | The brightly lit region is the upper part of the window. | 1 / 1 |
| 7 | The relevant part is at the top of the image, from the top of the window to the top of the curtain. | 1 / 1 |
| 8 | Locate the region labeled “window” and output coordinates [100,100,900,900]. | 1 / 1 |
Decoding shifts from a general window description to a specific region. Both representations improve the correct crop’s rank from 2 to 1 at step 4.
Case 3: Recognizing a Basset Hound — ImageNet-1K, sample 276
Correct class: Basset Hound, retrieved from all 1,000 classes.
| Step | Decoded output | Correct-class rank (e/z) |
|---|---|---|
| 1 | A Golden Retriever and a Beagle stand on the grass. | 2 / 1 |
| 3 | A Golden Retriever and a Sussex Spaniel stand outdoors. | 2 / 1 |
| 4 | A Golden Retriever and a Basset Hound stand outdoors. | 1 / 1 |
| 8 | The image is classified as containing a Basset Hound. | 1 / 1 |
Decoding shifts from incorrect breeds to the correct breed. When “Basset Hound” appears at step 4, the correct-class rank for e also improves from 2 to 1.
Case 4: Identifying the first celestial body shown — Video-MME, sample 136
Question: Which celestial body is shown first in the video?
Correct answer: Earth, among four candidates.
| Step | Decoded output | Correct-answer rank (e/z) |
|---|---|---|
| 1 | Earth viewed from space, in the foreground. | 2 / 2 |
| 2 | An animation of Earth focusing on its surface and atmosphere. | 2 / 1 |
| 5 | Earth viewed from space, with a largely unchanged scene. | 1 / 1 |
| 6 | The first celestial body shown in the video is Earth. | 1 / 1 |
| 8 | The first celestial body shown in the video is Earth. | 1 / 1 |
The trajectory-fused representation (z) reaches rank 1 at step 2, and the embedding-token representation (e) at step 5. Later decoded outputs directly answer the question.
Case 5: Retrieving a document containing an order number — ViDoRe_docvqa, sample 19
Question: What is the order number?
Relevant document ID: 19, retrieved from the full collection of 500 document pages.
| Step | Decoded output | Relevant-document rank (e/z) |
|---|---|---|
| 1 | A stack of blue and white shipping containers. | 3 / 7 |
| 2 | Packaging boxes with barcodes and product labels. | 2 / 6 |
| 5 | A product order form with an order-number field labeled “No.” | 1 / 3 |
| 6 | A black office chair with a red seat cushion. | 1 / 2 |
| 7 | A document or form with an “order no.” field. | 1 / 3 |
| 8 | A document or form containing “Order No. 40549”. | 1 / 2 |
(1) Response to Reviewer iqAg W1/W2 — Reconstruction Targets, Shared-Text Alignment, and the Role of SFT Supervision in VLR
We thank Reviewer iqAg for the constructive feedback.
W1 — Reconstruction Targets, SDR References, and Shared-Text Alignment
Each side uses its own final
answerfield from the Embed-RL[1] training annotations, containing an answer, description, or explanation. Five abbreviated examples follow:Across 1,471,320 SFT examples, only 0.0103% of pairs have identical reconstruction targets. Different texts can still share semantics: this intentionally promotes positive-pair alignment, consistent with contrastive learning. The question is what trajectory exploration adds beyond this supervision. Table 3, supplemented with MRMR and SDR-only RL results, shows:
Reconstruction improves deterministic AR by 0.8 points on MMEB-V2 and 0.9 on MRMR, supporting its alignment role and the reviewer’s explanation. SDR-only RL further improves VLR + Latent SFT by 0.4 and 2.4 points, respectively. The larger gain on reasoning-intensive retrieval supports our motivation: semantic decoding feedback guides exploration toward paths that better preserve task-relevant semantics and improve retrieval.
W2 — Which SFT Signals Make VLR Effective?
The supervision signals serve complementary roles:
In Table 3, VLR with only LNCEe and no direct trajectory supervision reduces performance from 63.8 to 63.1. Latent SFT raises it to 64.9, supporting the need for supervision to make exploration useful. Latent SFT also improves deterministic AR from 63.8 to 64.6.
Table 4 incorrectly uses the post-RL score of 65.7 as its reference, although all loss-ablation variants were evaluated before RL. We will correct the SFT-only reference to 64.9:
Removing LNCEZ raises the SFT score to 65.3, but subsequent RL under the same settings reaches only 65.1 versus the full model’s 65.7. This suggests that its main benefit here is supporting subsequent trajectory optimization rather than immediate SFT performance.
The w/o Lans variant undergoes only SFT, with no RL or SDR. Thus, no decoder lacking reconstruction training is used to compute SDR.
[1] Jiang et al. Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings. ECCV 2026.