- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EviAlign jointly designs the organization of generated evidence and the readout locations used to construct multimodal embeddings. The model generates Entity, Attribute, Relation, Detail, and Summary evidence, reads the final-layer state at the end of each unit, and averages and normalizes these states into a single retrieval vector. Training combines a bidirectional contrastive loss with generation losses on both the query and candidate sides. At inference, evidence is generated independently for each input, allowing candidates to be encoded offline. The paper trains on 500K pairs and evaluates twelve MMEB retrieval tasks, with a factorial study of evidence organization and readout location, readout ablations, and cost analysis.
Critical strengths
- The design gives evidence a concrete role in representation construction. The end of each semantic unit determines a readout location, connecting generated content to the embedding. The result remains a single vector, with independent query and candidate encoding and standard indexing and similarity computation.
- Controls using the same backbone support the configuration's effectiveness. In Table 3, semantic evidence and free-form CoT perform almost identically with a trailing readout, at 73.74 and 73.75, whereas semantic boundary readout reaches 76.94. Five trailing readouts do not reproduce this gain. Table 2 further controls readout count and pooling, showing that additional readout tokens alone do not explain the result.
Critical weaknesses
- The central interaction does not fully separate semantic correspondence from changes in generation. Mixed changes both unit order and the causal prefix available at each boundary. Distributed requires learning readout positions defined by the complete target length. All six models are trained separately and generate their own sequences, so matched training spans do not imply matched inference content. The interaction in Table 2, , is correct, but cannot yet be attributed entirely to the correspondence between semantic roles and boundaries. The 0.14 difference under Trailing weakens the explanation of a general decline in generation quality, but does not remove these remaining confounds. A single seed also leaves the interaction's stability uncertain. Training readouts additionally use teacher forcing, whereas inference relies on the student's own evidence. Appendix C.4 evaluates training-target quality, and the 99.52% validity rate measures query-side boundary formatting for the final model. Neither directly establishes the reliability of the student's generated content. This limits the evidence for robustness of the proposed interface.
- The incremental benefit of generating all five units has not been adequately weighed against its cost. Using only ENT from the same five-readout checkpoint already reaches 76.12, compared with 76.94 for the full aggregate. In a causal model, subsequent generation does not change the ENT state already computed, making early stopping at that boundary a direct cost control. A separately trained single-unit model is not a substitute. Table 13 reports 1,502.5 ms for a batch of four queries, versus 94.6 ms for direct encoding, a roughly 15.9-fold difference; these are not single-query latencies. TTE already uses generated context to improve embeddings [1]. The paper should clarify the quality–cost advantage of the full boundary scheme over early stopping and matched generation-based baselines. The current evidence supports effectiveness, but does not sufficiently establish the need for the additional generation. [1] Think Then Embed: Generative Context Improves Multimodal Embedding, International Conference on Learning Representations (ICLR), 2026.
Questions for authors
- Mechanism and stability: What are the multi-seed results, query- and candidate-side format validity rates, and generation lengths for the six configurations? Could a fixed global permutation or controlled readout analysis on shared evidence help separate semantic correspondence from order and content differences?
- Evidence reliability: How do entity, relation, and detail errors in the student's generated evidence relate to retrieval failures? How large is the gap between teacher-forced and freely generated readouts, both in distribution and out of distribution?
- Minimum useful computation: With the same five-readout checkpoint, what are the quality, generated token count, and runtime when stopping at ENT, ATT, or later boundaries? Are the gains from all five readouts concentrated in particular tasks?
- Fair cost comparison: With the same student, training pairs, and timing protocol, how does boundary readout compare with generation-based embedding baselines in quality and cost? Beyond the 500K pairs, what are the query- and candidate-side teacher supervision token counts, training cost, and offline candidate-encoding cost?