acceptodds
Under review as a conference paper at ICLR 2027

Learning Multimodal Embeddings with Evidence-Aligned Readout

Abstract

Multimodal large language models can expose task-relevant evidence through generation, but producing useful evidence does not by itself determine how it enters a retrieval embedding. We study whether the semantic organization of that evidence can also specify where representations are read. To address this question, we introduce **EviAlign**, which couples *Semantic Evidence Generation* with *Boundary Readout* in a shared multimodal large language model. It organizes evidence into five semantic units, reads the contextualized state at each unit boundary, and aggregates these states into a single normalized embedding. Generation and contrastive retrieval objectives jointly train this shared structure. With the same trailing readout, semantic evidence and free-form CoT yield nearly identical retrieval performance, suggesting that evidence organization alone does not explain the full gain. A controlled 2 × 3 study compares consistent and permuted evidence organization across three readout strategies, using training targets with matched evidence spans. With five readout states and the same mean pooling, the advantage of consistent semantic organization grows from 0.65 points at length-based training positions to 2.39 at evidence boundaries, yielding a 1.74-point co-design interaction. Across 12 MMEB retrieval tasks, EviAlign achieves 76.9 average Recall@1 with 500K training pairs while retaining single-vector indexing and scoring.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Retrieval, Multimodal Representation Learning, Multimodal Large Language Models, Multimodal Embeddings

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_eT24OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EviAlign jointly designs the organization of generated evidence and the readout locations used to construct multimodal embeddings. The model generates Entity, Attribute, Relation, Detail, and Summary evidence, reads the final-layer state at the end of each unit, and averages and normalizes these states into a single retrieval vector. Training combines a bidirectional contrastive loss with generation losses on both the query and candidate sides. At inference, evidence is generated independently for each input, allowing candidates to be encoded offline. The paper trains on 500K pairs and evaluates twelve MMEB retrieval tasks, with a factorial study of evidence organization and readout location, readout ablations, and cost analysis.

Critical strengths
  1. The design gives evidence a concrete role in representation construction. The end of each semantic unit determines a readout location, connecting generated content to the embedding. The result remains a single vector, with independent query and candidate encoding and standard indexing and similarity computation.
  2. Controls using the same backbone support the configuration's effectiveness. In Table 3, semantic evidence and free-form CoT perform almost identically with a trailing readout, at 73.74 and 73.75, whereas semantic boundary readout reaches 76.94. Five trailing readouts do not reproduce this gain. Table 2 further controls readout count and pooling, showing that additional readout tokens alone do not explain the result.
Critical weaknesses
  1. The central interaction does not fully separate semantic correspondence from changes in generation. Mixed changes both unit order and the causal prefix available at each boundary. Distributed requires learning readout positions defined by the complete target length. All six models are trained separately and generate their own sequences, so matched training spans do not imply matched inference content. The interaction in Table 2, (76.94−74.55)−(74.85−74.20)=1.74(76.94-74.55)-(74.85-74.20)=1.74, is correct, but cannot yet be attributed entirely to the correspondence between semantic roles and boundaries. The 0.14 difference under Trailing weakens the explanation of a general decline in generation quality, but does not remove these remaining confounds. A single seed also leaves the interaction's stability uncertain. Training readouts additionally use teacher forcing, whereas inference relies on the student's own evidence. Appendix C.4 evaluates training-target quality, and the 99.52% validity rate measures query-side boundary formatting for the final model. Neither directly establishes the reliability of the student's generated content. This limits the evidence for robustness of the proposed interface.
  2. The incremental benefit of generating all five units has not been adequately weighed against its cost. Using only ENT from the same five-readout checkpoint already reaches 76.12, compared with 76.94 for the full aggregate. In a causal model, subsequent generation does not change the ENT state already computed, making early stopping at that boundary a direct cost control. A separately trained single-unit model is not a substitute. Table 13 reports 1,502.5 ms for a batch of four queries, versus 94.6 ms for direct encoding, a roughly 15.9-fold difference; these are not single-query latencies. TTE already uses generated context to improve embeddings [1]. The paper should clarify the quality–cost advantage of the full boundary scheme over early stopping and matched generation-based baselines. The current evidence supports effectiveness, but does not sufficiently establish the need for the additional generation. [1] Think Then Embed: Generative Context Improves Multimodal Embedding, International Conference on Learning Representations (ICLR), 2026.
Questions for authors
  1. Mechanism and stability: What are the multi-seed results, query- and candidate-side format validity rates, and generation lengths for the six configurations? Could a fixed global permutation or controlled readout analysis on shared evidence help separate semantic correspondence from order and content differences?
  2. Evidence reliability: How do entity, relation, and detail errors in the student's generated evidence relate to retrieval failures? How large is the gap between teacher-forced and freely generated readouts, both in distribution and out of distribution?
  3. Minimum useful computation: With the same five-readout checkpoint, what are the quality, generated token count, and runtime when stopping at ENT, ATT, or later boundaries? Are the gains from all five readouts concentrated in particular tasks?
  4. Fair cost comparison: With the same student, training pairs, and timing protocol, how does boundary readout compare with generation-based embedding baselines in quality and cost? Beyond the 500K pairs, what are the query- and candidate-side teacher supervision token counts, training cost, and offline candidate-encoding cost?
Reviewer_3aQTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviAlign, a multimodal embedding framework that integrates structured evidence generation with boundary-aligned representation readout. The method first uses an MLLM to generate five semantic evidence units (entity, attribute, relation, detail, and summary) for each training input, then trains another MLLM using these generated targets with both a language modeling loss and a contrastive learning loss. For the contrastive learning loss, the method extracts the five hidden states at their corresponding boundaries (corresponding to each semantic evidence unit) and aggregates them into a single embedding with mean pooling. Experiments on 12 MMEB retrieval tasks show competitive performance, and controlled comparisons suggest that aligning readout positions with semantic evidence boundaries is more effective than conventional trailing or distributed readout strategies.

Critical strengths
  1. The paper presents a simple and intuitive approach to integrating structured evidence generation with multimodal embedding learning. The idea of aligning representation readout with generated semantic units is interesting and potentially useful.
  2. The paper provides reasonably comprehensive ablation studies, particularly the controlled comparisons between evidence organization and readout strategies. These experiments offer useful empirical observations about how representations can be constructed from generated evidence.
Critical weaknesses
  1. InfoNCE encourages a more uniform embedding space, while the pretrained language modeling head is adapted to potentially highly anisotropic hidden representations. One concern is that joint optimization may introduce conflicting geometric constraints, particularly for vocabulary tokens receiving limited supervision during fine-tuning. It would be useful to analyze the impact on representation geometry and generation quality, and whether loss weighting or a separate projection head could mitigate potential interference.
  2. Due to causal attention, earlier evidence units can influence more subsequent boundary representations than later units. Therefore, mean pooling over the five hidden states does not necessarily give equal importance to the five semantic units. Since their order is fixed, semantic importance and positional effects are also difficult to disentangle. Experiments with alternative evidence orderings and analyses of individual unit contributions would be helpful.
  3. The “Mixed” setting randomly permutes the evidence units for each training example, whereas the “Semantic” setting maintains a consistent order. This could make generation more difficult for Mixed, independently of semantic boundary alignment. Therefore, the performance gap may partly reflect differences in generation difficulty rather than the proposed alignment mechanism. A comparison that controls for evidence ordering while varying semantic-role-to-boundary correspondence would strengthen the conclusions.
Questions for authors

see above.

Reviewer_U9a4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EviAlign uses structured evidence generation to guide multimodal embedding construction. It generates five semantic units, reads the hidden states at their boundaries, and pools them into a single retrieval vector. The model is trained jointly for generation and retrieval. Experiments on 12 MMEB retrieval tasks, together with ablations of evidence organization and readout strategy, support the benefit of coupling these two components.

Critical strengths
  1. The paper asks a useful, focused question about where to read representations after evidence generation. The ablations go beyond a simple baseline comparison and show that structured generation alone or extra readout tokens do not explain the full improvement.

  2. The method retains a single-vector retrieval interface and is evaluated across varied tasks with both Qwen2-VL and Qwen3-VL. This makes the contribution relevant to existing dense-retrieval systems.

Critical weaknesses
  1. The interpretation of the central comparison needs more support. Although the training evidence is matched, the six configurations generate their own evidence at evaluation. The sample-specific shuffling in Mixed may also affect generation difficulty and quality. The results support the combined method, but do not fully establish whether the 1.74-point interaction reflects stable role-to-boundary correspondence or, in part, differences in the generated evidence (Section 4.3; Appendix B.6).

  2. The accuracy–cost tradeoff of generating all five units at inference is not fully established. ENT alone achieves 76.12, compared with 76.94 for the full aggregate (Table 8). Although Table 13 reports substantial generation overhead relative to direct encoding, it does not measure the cost of stopping at ENT. This leaves unclear whether the incremental accuracy justifies generating the remaining four units.

Questions for authors
  1. How do generation quality and format validity differ across the six factorial configurations? As a diagnostic, evaluating the interaction with matched evidence content would help assess whether differences in generation contribute to the result.

  2. Could you provide per-task results for the factorial comparison, showing whether the interaction is consistent across tasks or concentrated in a few datasets?

  3. For the ENT-only setting in Table 8, what query encoding time would stopping immediately after computing the ENT state achieve under the measurement setup of Table 13, using ENT embeddings for both queries and candidates?

Reviewer_NUP8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how evidence generated by MLLMs should be incorporated into embeddings and proposes evidence-aligned readout, based on the idea that the semantic structure of generated evidence should also determine where the embedding is read from.

  • Semantic Evidence Generation: For each input, the model generates structured evidence in five categories: Entity, Attribute, Relation, Detail, and Summary. Each category ends with a boundary token (<ENT>, <ATT>, <REL>, <DET>, <SUM>). The training targets are generated using GLM-4.1V-9B-Thinking.

  • Boundary Readout: The final-layer hidden states of the five boundary tokens are averaged and normalized to produce a single-vector embedding. Both queries and candidates are generated and encoded using the same procedure.

  • Training: The model jointly optimizes the language modeling loss and a symmetric InfoNCE loss.

The central evidence comes from a controlled 2×32 \times 3 experiment combining evidence composition (Semantic vs. Mixed) with readout strategies (Trailing, Distributed, and Boundary). The authors report a 2.39-point Semantic–Mixed performance gap with Boundary Readout, compared with 0.65 points with Distributed Readout, claiming an interaction effect of 1.74 points.

Across 12 retrieval tasks in MMEB, the method achieves an average Recall@1 of 76.9 using a Qwen3-VL-8B backbone trained on 500,000 pairs.

Critical strengths
  1. Controlled experiments that isolate individual design components. The authors conduct a 2×3 experiment that varies evidence composition and readout position while keeping the same evidence spans.

Additional baselines include no generation with multiple readouts (70.30), free-form CoT with trailing readout (73.75), structured evidence with trailing readout (73.74), and adding five readout tokens at the end (MLTR, 73.07). These results demonstrate that the gains cannot be reproduced simply by adding generation or increasing the number of readout tokens. The 8.6-point improvement over the direct embedding baseline (68.30), trained using the same backbone and data, is also substantial.

  1. Preserving the single-vector interface and practical insights. The method retains a single vector per document, preserving the conventional retrieval interface. Moreover, prepending free-form CoT increases the number of generated tokens by approximately sixfold (702 vs. 114) without improving performance (76.59 vs. 76.94). This is a useful finding regarding the efficiency of using concise, structured evidence.
Critical weaknesses

Overall, I view the idea and experimental design positively. However, all experiments are conducted on Qwen-family backbones (Qwen2-VL and Qwen3-VL), which share similar architectures and training recipes. To show that evidence-aligned readout generalizes beyond this family, I encourage the authors to evaluate it on backbones with different architectures, such as InternVL or LLaVA-style models, as well as newer versions such as Qwen3.5. This is important for assessing whether the method holds when the underlying architecture changes, which directly bears on its applicability in real-world settings.

In addition, although Figure 3a reports overall results for 2B backbones, the controlled analyses (the 2 x 3 study and the decomposition in Table 3) are conducted only at 8B. It is therefore unclear whether boundary readout provides the same advantage over direct embedding and trailing readout at smaller scales. Since the method relies on generating all five boundary tokens correctly, I would also like to see format-validity rates for smaller models, as Appendix C.7 suggests that missing boundary tokens can severely degrade performance. Results on even smaller backbones (e.g., 1B or below) would further clarify whether the approach remains practical in resource-constrained deployments, where its generation cost matters most.

Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.