acceptodds
Under review as a conference paper at ICLR 2027

TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens

Abstract

Recent research has demonstrated that Universal Multimodal Embedding (UME) benefits significantly from Chain-of-Thought (CoT) reasoning. In this paradigm, a generative model produces explicit reasoning traces for a multimodal query, with the final representation extracted from an <eos> embedding token attending to both the query and the reasoning. Despite its effectiveness, the computational overhead of generating explicit CoT traces is often prohibitive. In this work, we propose replacing explicit CoT with latent think tokens, which are interpreted as latent variables that can produce explicit CoT traces as observed variables. By optimizing think tokens using CoT generation loss and subsequent embedding tokens using contrastive loss, we produce high-performance, reasoning-aware representations at a constant inference cost. Our study investigates two key architectural designs: 1) how think and embeddings tokens should be extracted from the same LLM backbone. 2) how the tokens should be trained as two dependent tasks. We introduce TTE-Flash-2B, a reasoning-aware multimodal representation model that outperforms its explicit-CoT counterpart on the MMEB-v2 benchmark, while producing latent think tokens that are interpretable both textually and visually. Furthermore, zero-shot evaluation across 15 video datasets reveals scaling behavior as the number of think tokens increases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Think-Then-Embed, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FqTROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TTE-Flash, which replaces the explicit CoT in Think-Then-Embed with a fixed number of latent think tokens. Within one MLLM backbone the method produces two kinds of continuous tokens, think and embed; it compares loop-based and register-based extraction and adopts per-layer registers. For training, the think hidden vectors serve as an information bottleneck from which a frozen LLM reconstructs the explicit CoT, embed tokens are supervised with a contrastive loss, and the two use decoupled positional parameters. With Qwen3-VL-2B, MMEB-V2 overall is 64.1, above the explicit-CoT TTEs-2B (63.1), with about 70× the throughput. The paper also analyzes think-token scaling, interventions, CoT form, visualization, and adaptive budgets.

Critical strengths

A clear and practical efficiency–accuracy analysis: Table 1 compares loops and registers within one framework (8 registers: 71 ms, 14.1 sample/s; 8 loops: 179 ms, 5.6 sample/s; PLUME: 298 ms, 3.3 sample/s; TTEs: about 5k ms, 0.2 sample/s), and offers a low-cost per-layer-register improvement, which is directly valuable for retrieval systems with strict latency constraints.

A simple training objective with fairly thorough behavioral analysis: Think tokens act as a bottleneck for CoT generation, supervised by a frozen LLM decoder; masking/shuffling/noising the think states clearly degrades performance, and increasing the number of think tokens helps more steadily on reasoning-heavy zero-shot video tasks (ActivityNet-QA, NExT-QA, VideoMME), suggesting the think states are genuinely used by the embedding.

Critical weaknesses

"Outperforming explicit CoT" comes mainly from the video sub-score; image and document sub-scores still lag, and Table 2 contains a numerical inconsistency. In Table 2, TTE-Flash's Image overall of 68.3 is below TTEs' 70.1 (QA 61.7 vs. 66.6, GD 81.3 vs. 84.1), and VisDoc 68.1 is below 68.8; the overall advantage comes from video. However, TTEs' video All column is 32.1, below all four of its video sub-columns (47.3, 49.1, 34.4, 33.2), which is arithmetically impossible for a weighted average; weighting by the number of MMEB-V2 datasets (36/18/24, 78 in total) reproduces TTE-Flash's 64.1 and UME-R1's 60.1, whereas plugging in the table's 32.1 with the same weights gives about 60.9 for TTEs, and obtaining 63.1 would require a video average of about 41.5. This cell may therefore be wrong, and the size of the video advantage needs to be verified. At the same scale, appendix Table 7 lists Embed-RL-2B at 66.8 and Qwen3-VL-Embedding-2B at 73.4, while TTE-Flash is compared only against "open-data" models, with differences in training data and backbone left unquantified.

Key design conclusions lack numeric and statistical support, and the "bottleneck" interpretation is loose. Loop > register (about 3% relative gap), per-layer registers narrowing the gap, pairwise > sum-of-max ("comparable performance"), and decoupling > sharing are mainly shown as trends in figures (averaged over 20 checkpoints), while the main table and other analyses are single-run results; none has confidence intervals. The effects of the number of think tokens and embed tokens are confounded (with think=8, adding embed tokens saturates). The "information bottleneck" is only an analogy, with no capacity/mutual-information measurement; caption supervision is almost as good as full CoT (70.4 vs. 70.8), suggesting that think tokens encode descriptive semantics rather than reasoning.

Questions for authors

Is the comparison with TTEs/UME-R1/PLUME done under the same training data, backbone and number of epochs? Can you provide an explicit-CoT baseline on the same backbone?

Please check the TTEs video All (32.1) and overall 63.1 in Table 2, and explain where the video improvement comes from (video training data, number of think tokens).

Please turn the key results of Figs. 4–9 into tables with mean ± standard deviation, and provide multi-seed variance for the main table.

For the 70× comparison, does TTEs' 5k ms use inference acceleration (e.g., vLLM)? What is TTE-Flash's latency with long inputs (video, visual documents) and larger batches?

What happens in a control with the generation loss removed and only registers kept (same parameter count), to demonstrate the value of generation supervision?

Reviewer_vL2BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TTE-Flash replaces explicit reasoning text with learned latent think tokens. A Qwen3-VL-2B backbone produces think states followed by retrieval embeddings. During training, the think states condition the frozen, adapter-free backbone to reconstruct CoT traces, while embedding states receive contrastive supervision. The preferred design uses parallel per-layer registers instead of sequential latent loops. Its final configuration has 32 think tokens and one embedding token in a single prefill pass, reaching 64.1 on MMEB-V2 versus 63.1 for the explicit-CoT TTE student and 61.6 for PLUME.

Critical strengths
  1. The paper is well motivated, and some design choices are interesting. For example, the authors use a CoT reconstruction objective to provide direct supervision for the latent states, while parallel registers replace slow autoregressive generation.

  2. The evaluation is comprehensive, and the method achieves substantial speedups. The architectural study is also informative. It examines loops versus registers, per-layer parameters, token decoupling, similarity functions, and token budgets, which may be helpful to the community.

Critical weaknesses
  1. The main accuracy gain is not adequately separated from differences in backbone and supervision. TTE-Flash uses Qwen3-VL-2B, whereas TTE and PLUME use Qwen2-VL, and the CoT data are not matched.

  2. I would appreciate further discussion of the necessity of latent reasoning. The paper shows that masking or randomly replacing think states reduces performance. However, these interventions seem to demonstrate more directly that this information channel is useful, rather than establishing that it carries a specific, interpretable reasoning process. On MSCOCO, caption supervision also performs nearly as well as CoT supervision (70.4 versus 70.8). This suggests that a substantial portion of the gain may come from reconstructing semantic information, while the contribution of reasoning specific to CoT remains unclear.

Similarly, decoding text or images from think states demonstrates that they contain recoverable semantic information. However, the decoded examples still include unsupported details, and the paper does not establish that these outputs faithfully explain the final retrieval decisions. It would therefore be helpful to distinguish the presence of semantic information from evidence that latent reasoning is necessary.

Questions for authors

The visual presentation of the figures and tables could be further improved.

Reviewer_KuYwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTE-Flash, a latent-reasoning approach for universal multimodal embeddings. It adds learned think and embed registers to a causal multimodal language model and uses explicit CoT to supervise the think registers. At inference, the hidden states of these registers are extracted in a single prefill pass, substantially reducing reasoning latency. The paper compares loop-based and parallel register-based designs and introduces per-transformer-layer registers to narrow the accuracy gap between latent reasoning and loop-based computation. With these design choices, the 2B model reports 64.1 on MMEB-V2, higher than the TTEs and PLUME scores reported in Table 2. The paper also reports a 70x inference-efficiency improvement over explicit-CoT TTEs in its Table 1 comparison.

Critical strengths
  • TTE-Flash addresses the inference cost of explicit CoT generation for retrieval. Its register-based design extracts think and embed states in a single prefill pass, offering a plausible route to lower latency while retaining intermediate representations for embedding.
  • The evaluation spans image, video, and visual-document retrieval on MMEB-V1/V2 and includes analyses of token budgets, CoT targets, and test-time think-state perturbations. The paper reports a 64.1 MMEB-V2 score for its 2B model and provides useful empirical coverage of how the proposed system behaves across tasks and settings.
Critical weaknesses
  • The core technical contribution appears incremental. TTE-Flash's main contribution is to use explicit CoT, decoded by an LLM, to supervise latent think tokens. Explicit-to-latent transfer and CoT distillation are established directions, and PLUME has already applied latent reasoning to UME using explicit CoT as a training scaffold. TTE-Flash's distinctive implementation combines CoT reconstruction through a frozen decoder with parallel/per-layer think registers and separate embed registers. This may be a useful task-specific integration, but the paper does not establish a sufficiently distinct reasoning principle, which limits the method-level novelty and technical contribution. The authors should clarify what conceptual advance this design provides beyond prior CoT-supervised latent reasoning.
  • The experiments do not isolate the effectiveness of TTE-Flash's CoT-supervised latent-reasoning method or explain its advantage over related approaches. In particular:
    1. Table 2 reports 64.1 for TTE-Flash, compared with 61.6 for PLUME and 63.1 for TTEs, but this is not a controlled method comparison. TTE-Flash uses Qwen3-VL-2B, whereas PLUME's reported MMEB-V2 result uses Qwen2-VL-2B, and their training recipes differ. Table 1's latency results also use different configurations: four registers for TTE-Flash, eight reasoning tokens for PLUME, and 300 generated tokens for explicit TTEs. These cross-paper results show the performance of complete systems but do not establish the incremental effectiveness of TTE-Flash's training method or register design. A comparison with matched backbones, data, and training settings is needed.
    2. The CoT-reconstruction objective is not isolated. Table 4 compares different CoT targets but includes no matched model trained without the reconstruction loss. Table 3's masking and shuffling interventions show that retrieval uses the learned think states, but they do not establish that CoT supervision, rather than the addition of learned registers or other training effects, causes the gain.
    3. The reported results also leave an unexplained gap between the architecture ablation and the full benchmark. Figure 4 shows that on MMEB-V1 with a total token budget of 8, loop-based models outperform register-based models, while per-layer registers narrow the gap. Table 2 reports that the 32-think/1-embed TTE-Flash model scores 64.1 on MMEB-V2, above explicit-CoT TTEs at 63.1. Since these results use different benchmarks and configurations, they are not a direct contradiction, but the paper does not identify which changes account for the relative-performance reversal. Please provide matched ablations or analysis that separates the effects of CoT supervision, register architecture, token budget, and training recipe.
Questions for authors
  1. How are the per-transformer-layer register tokens implemented in the forward pass? Are layer-specific register parameters applied to fixed register positions, or are new register positions appended at each layer? Please clarify how register hidden states are propagated or combined across layers and how their positional encodings are handled, ideally with tensor shapes or pseudocode.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.