- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ALICE, a multimodal LLM embedder that inserts 15 learnable latent tokens between the query and the embedding token. The goal is to keep the gains of chain-of-thought (CoT) embedders without generating text at inference.
Training is split into 3 stages:
- Qwen3-VL-235B writes a 5-step CoT for each MMEB training pair, with the target visible. A reference embedder then learns to map the full CoT and each single step to the target embedding.
- The latent tokens are mean-pooled into 5 groups, and each group is aligned with InfoNCE to the embedding of its step. A query-to-target contrastive loss is trained alongside.
- The alignment loss is dropped, and training continues with the contrastive loss alone.
The approach is motivated by an initial analysis. Latents trained with a COCONUT curriculum decode mostly to tokens that start CoT steps. MMEB scores each task by Hit@1, the share of queries whose correct target ranks first among the candidates. On MMEB, ALICE reaches 70.6 to 74.8 Hit@1 across seven Qwen-VL backbones from 2B to 8B. Analyses of a 5-token variant are considered to support reasoning in latent space.
I currently consider this paper to be on the fence, cutting right between acceptance and rejection. The novelty is generally limited in the crowded field, but the experiments do add value. However, such a work needs to pass high quality bar for the experimental setup that this work does not quite yet meet (cf. weaknesses). I invite the authors to rebut my review.
Critical strengths
- Relevance: Embedders that generate a CoT before embedding are slow at query time. Table 2 reports 2,018 ms per query for RGE and 7,229 ms for UME-R1, vs. 6.7 ms for ALICE. ALICE gets close to or above these methods on MMEB with a single forward pass.
- Broad baseline coverage: Results cover 7 backbones from 3 Qwen-VL generations.
- Solid experimental ablations: Aligning the latents to CoTs from other instances scores 63.0, vs. 63.9 without alignment (Table 8). Aligning them to a generic task specification scores 64.0. Both results show that the useful signal is instance-specific.
Critical weaknesses
The authors claim that the latent tokens carry reasoning distilled from the CoT. Tables 3 and 5 do not separate the latent tokens from the teacher signal. Specifically:
- Unaligned latents score 66.0 vs. 65.8 without latents.
- Aligning the latents to the plain target embedding, excl. CoT, reaches 70.2. Full ALICE reaches 71.9.
- Most of the alignment gain thus appears without CoT steps. The CoT-specific part is 1.7 points from a single run.
The missing variant distills the same reference-embedder signal into the embedding token, excl. latents, at matched epochs. An epoch-matched baseline is also missing: ALICE trains the embedder for 2 contrastive epochs, while the baseline trains for one.
Experimental setup: All results come from single runs. The paper describes no validation split & every ablation reports MMEB test scores. After the latent-budget sweep in Fig. 4, the paper sets the best-scoring budget as the default (Sec. 4.3). If design choices were made on these scores, the reported gains are overstated.
The most comparable baselines were not evaluated:
- TTE is cited and appears in Table 7, but its MMEB-v1 results are absent from Table 1. With Qwen2-VL-7B, TTE reaches 73.3 with a small trained reasoner and 78.5 with a 72B reasoner.
- MMEmb-R1 uses the same Qwen-VL backbones and reports 74.8 on the 36 image tasks at 4B, vs. 72.8 for ALICE.
- Single-pass latent-token embedders such as LaME and TTE-Flash are the closest competitors to the efficiency claim. Both report MMEB-v2 results and should be compared or at least discussed.
- Only Qwen2.5-VL-3B has an in-house baseline trained with ALICE's recipe. For the other backbones, gains over copied numbers mix the method with the recipe and the data.
Questions for authors
- What does a model reach that distills the reference-embedder signal into the embedding token, without latents, at equal training budget? An example is an auxiliary loss that aligns the embedding token to the CoT embedding or to the final-step embedding.
- Which training data produced each ALICE row in Table 1? The Qwen2-VL-2B and Qwen2-VL-7B rows match the image results of the MMEB-v2 models in Tables 7 and 11 in every column, while the other backbones differ. Sec. 4.1 states that training uses the MMEB training split. If these two rows come from the 1.8M-example model, what do image-only models reach?
- How close is each latent group to its step anchor after the free-form stage? It would be interesting to assess the cosine similarity, or the accuracy of retrieving the correct step among the steps of the same instance.
- How were the number of latents, the objective, and the alignment target chosen? Did you use data held out from the MMEB test sets? Moreover, results of Tables 3 and 5 and for Fig. 4 would be ideally supported by 3 seeds.
- Does ALICE help on reasoning-intensive retrieval benchmarks such as MRMR, where a reasoning benefit should be largest?
10 comments
Rebuttal to Reviewer nSah (2/3)
Q3: How close is each latent group to its step anchor after the free-form stage?
We thank the reviewer for this suggestion. We assess the correspondence between latent groups and CoT anchors through within-instance step matching. We divide the 15 latent tokens into five groups and mean-pool each group, then compute cosine similarities between each group representation and the five CoT step anchors encoded by the fixed reference embedder. Matching accuracy is the proportion of cases in which the originally corresponding step ranks first.
To further examine retention of step correspondence, we sample 200 examples from each training subset, for a total of 4,000 examples, and compare S2 and S3 on the same examples:
| Sample scope | After S2 | After S3 |
|---|---|---|
| 200 samples/subset * 20 subsets | 59.92% | 54.60% |
Step correspondence remains observable after S3, with a matching accuracy of 54.60%, compared with 59.92% after S2. Our free-form training is inspired by latent-reasoning methods such as COCONUT, which shift to final-task supervision in the last training stage. In ALICE, S2 provides step-level semantic supervision, while S3 removes the alignment loss and allows the intermediate representations to adapt to the retrieval objective. In the stage ablation in Table 3, adding S3 improves overall retrieval performance from 69.7 to 71.9. These results suggest that the intermediate representations remain associated with their corresponding CoT steps after adaptation to retrieval.
Q4: How were the number of latents, the objective, and the alignment target chosen? Results of Tables 3 and 5 and Fig. 4 would ideally be supported by 3 seeds.
We agree that experiments with multiple seeds help assess stability. We have completed three-seed experiments for the four ALICE latent-budget configurations with relatively close scores in Fig. 4 and added means and standard-deviation error bars to the revised figure. We also provide three-seed results for ALICE with Qwen2-VL-2B on MMEB-v2, with an overall score of 64.99±0.01 (mean ± standard deviation). The complete results are included in Appendix B.2 / Table 9 of the revised manuscript and in our response to Reviewer FSQ3, Question 5.
Computational resources and the rebuttal timeline currently prevent us from completing three training seeds for every configuration in Tables 3 and 5. For these ablations, our reporting follows the practice of related work such as VLM2Vec, UME-R1, Embed-RL, PLUME, RGE, and CAFe, whose relevant ablations report point estimates without standard deviations across training seeds. The performance gaps in the main comparisons of these tables are larger, and we expect their conclusions to be less sensitive to training randomness.
Q5: Does ALICE help on reasoning-intensive retrieval benchmarks such as MRMR?
We thank the reviewer for this suggestion. We have added an evaluation of ALICE on the reasoning-intensive retrieval benchmark MRMR, covering all 11 tasks:
| Method | Backbone | Art | Med | Sci | Hum | Math | Phy | Eng | Bus | Neg | Design | Traffic | AVG(11) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OpenCLIP | ViT-G/14-1B | 56.0 | 17.9 | 33.2 | 22.0 | 5.7 | 5.0 | 7.0 | 9.7 | 13.0 | 8.1 | 12.4 | 17.3 |
| PLUME | Qwen2-VL-2B | 68.7 | 47.7 | 63.6 | 54.7 | 19.2 | 30.3 | 29.2 | 44.5 | 7.5 | 44.3 | 21.9 | 39.2 |
| UME-R1 | Qwen2-VL-2B | 70.6 | 48.8 | 64.9 | 58.0 | 20.9 | 30.0 | 26.8 | 46.7 | 7.0 | 42.6 | 27.8 | 40.4 |
| ALICE | Qwen2-VL-2B | 73.5 | 48.3 | 58.9 | 56.5 | 21.6 | 30.1 | 28.4 | 46.5 | 6.0 | 41.9 | 37.5 | 40.8 |
| VLM2Vec | Qwen2-VL-7B | 37.2 | 19.9 | 34.1 | 29.4 | 12.2 | 15.3 | 16.9 | 35.4 | 5.0 | 3.47 | 7.53 | 19.7 |
| UME-R1 | Qwen2-VL-7B | 77.8 | 55.7 | 72.9 | 64.1 | 27.2 | 39.2 | 32.2 | 47.8 | 7.5 | 61.9 | 41.7 | 48.0 |
| ALICE | Qwen2-VL-7B | 75.7 | 54.3 | 68.3 | 60.1 | 31.1 | 43.1 | 34.0 | 53.0 | 9.5 | 70.0 | 40.3 | 49.0 |
With Qwen2-VL-2B, ALICE achieves an average score of 40.8, compared with 40.4 for UME-R1 and 39.2 for PLUME. With Qwen2-VL-7B, ALICE achieves 49.0, compared with 48.0 for UME-R1. These results are included in Appendix B.3 / Table 10 of the revised manuscript.
Rebuttal to Reviewer nSah (3/3)
Additional response to weaknesses: Baseline comparisons and controlled experiments
We thank the reviewer for highlighting these related methods. We have expanded the comparisons in the revised manuscript. Table 1 now includes the available MMEB-v1 results for TTE, MMEmb-R1, and LaME, while Appendix B / Table 7 includes the MMEB-v2 comparisons with MMEmb-R1, LaME, and TTE-Flash. We have also expanded the discussion in the Related Work section.
We use Qwen2.5-VL-3B as a representative backbone for systematic ablations, as its model size makes it practical to evaluate a broad range of design choices under controlled conditions. By keeping the backbone and training data fixed and matching the training budget within each comparison, we can assess the contributions of ALICE's different designs. Our conclusions about these design choices are based on these controlled experiments. We then evaluate the complete method across seven backbones to examine whether its effectiveness extends to different model sizes and generations.
Rebuttal to Reviewer nSah (1/3)
Q1: What does a model reach that distills the reference-embedder signal into the embedding token, without latents, at equal training budget?
We thank the reviewer for this suggestion. We have added a control that uses no latent tokens and directly distills the CoT representation from the same reference embedder into the final embedding token. The experiment uses the 160K training data and Qwen2.5-VL-3B setting from Appendix C, with the same S2 training budget. All variants are evaluated after S2.
Specifically, the frozen reference embedder encodes the full CoT. A cosine loss aligns the final <emb> representation, computed without latent tokens, with the CoT representation produced by the reference embedder. We test two settings: cosine-loss distillation alone, and cosine-loss distillation combined with the query–target retrieval contrastive loss:
Direct CoT distillation combined with retrieval contrastive learning achieves 59.9, compared with 67.9 for ALICE. Our reference embedder is designed to convert CoT information into retrieval-relevant semantic anchors for intermediate supervision, rather than to provide a complete target representation for the final query embedding. In ALICE, these anchors guide the intermediate latent states, while the final embedding learns to integrate the query and latent information through the retrieval objective. The results support this use of CoT representations as intermediate supervision over directly distilling them into the final embedding. We have added this control and its description to Appendix C / Table 11 of the revised manuscript.
We also address the request for an epoch-matched baseline in the Weaknesses section: we train both the standard embedding baseline without latents and the baseline with the same number of latent tokens but no CoT alignment for two epochs, matching the combined training epochs of S2 and S3 in full ALICE.
After two epochs, the standard and latent-only baselines achieve overall scores of 66.7 and 67.0, respectively, compared with ALICE's 71.9. These results indicate that additional training epochs, with or without learnable latent tokens, are insufficient to explain ALICE's improvement.
Q2: Which training data produced each ALICE row in Table 1? If these two rows come from the 1.8M-example model, what do image-only models reach?
We thank the reviewer for pointing this out. The Qwen2-VL-2B and Qwen2-VL-7B rows in Table 1 mistakenly reported results from checkpoints trained on the MMEB-v2 mixture. We have replaced them with the corresponding results from models trained on the MMEB image-only data. All ALICE rows in the revised Table 1 now report results from models trained on the MMEB image-only training data. The original and corrected results are shown below:
The Overall score for Qwen2-VL-2B is updated from 71.1 to 71.3. For Qwen2-VL-7B, the Overall score remains 74.8 to one decimal place.
We sincerely apologize for this error and the resulting confusion. The correction affects only these two rows. The ablations analyzing the method's contributions in §4.3 and the relevant appendices all use Qwen2.5-VL-3B under controlled settings, so those ablations and their conclusions are unaffected.