acceptodds
Withdrawn from ICLR 2027

Pixel as the Common Language: Unified Multimodal Embedding via Visual Rendering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal embedding models map heterogeneous inputs into a shared representation space, yet text-only, image-only, and composed inputs are still presented in distinct formats and follow different routes through the model. This fragmentation prevents heterogeneous tasks from jointly optimizing the same end-to-end encoding process. To address this issue, we propose unification before encoding and instantiate this paradigm in RenderEmbed. The framework uses Visual Substrate Rendering to convert heterogeneous inputs into a common visual substrate and then encodes the resulting visual-token sequence with a VLM. To retain both broader context and fine-grained cues in compact retrieval representations, it applies Progressive Hierarchical Anchor Encoding by interleaving anchors throughout the visual-token sequence and fusing their representations across layers. It further employs Interventional Rendering Learning to distinguish retrieval semantics from presentation variations through complementary content and rendering interventions. Extensive experiments on MMEB and ViDoRe v2 show that RenderEmbed outperforms prior state-of-the-art models at the same scale and exhibits strong out-of-domain generalization. Compared with MetaEmbed, RenderEmbed improves MMEB Overall Precision@1 by 1.7 points, with a larger 2.1-point gain on out-of-domain tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Multimodal Embeddings, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hh9jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RenderEmbed, a unified multimodal embedding framework based on Unification before Encoding. The method converts text, images, and composed multimodal inputs into a common visual substrate, and employs Progressive Hierarchical Anchor Encoding and Interventional Rendering Learning to produce compact and robust multi-vector representations. Experiments on MMEB and ViDoRe v2 demonstrate competitive retrieval performance.

Critical strengths

1.Interesting research perspective. Unlike existing approaches that primarily focus on shared encoders or feature-space alignment, this paper explores unification at the input level, offering an interesting direction for unified multimodal representation learning.

2.The paper provides ablation studies on major components, anchor budgets, cross-layer fusion, and rendering robustness, helping to assess the effectiveness of different design choices.

Critical weaknesses

1.Unifying input formats does not necessarily lead to better cross-modal semantic alignment. Existing VLMs already integrate textual and visual information through shared backbones. The paper does not sufficiently explain why converting native text tokens into visual inputs should systematically improve representation learning, or whether this design generalizes to other models and tasks.

2.Although the complete RenderEmbed framework improves over NativeEmbed by 1.1 percentage points, the independent gain from visual input unification under standard InfoNCE is only 0.6 percentage points. The current experiments do not adequately demonstrate that the improvements arise from better knowledge sharing across heterogeneous modalities. Additional cross-modal transfer experiments and evaluations of pixel rendering on other multimodal retrieval architectures would strengthen this claim.

3.Rendering text into images may increase visual token counts and encoding overhead. However, the efficiency analysis mainly focuses on index storage and retrieval scoring, without fully accounting for rendering and encoding costs. Additional measurements of encoding latency, training throughput, and end-to-end retrieval latency are needed.

4.it remains unclear whether progressively interleaved anchors learn meaningful hierarchical semantic representations. Furthermore, the model exhibits noticeable performance degradation on certain visual grounding tasks, which warrants further investigation.

Questions for authors

1.Why should pixel-level unification outperform native multimodal inputs? Can the authors provide a clearer explanation or empirical evidence demonstrating its unique benefits and applicable conditions?

2.Do the performance improvements genuinely originate from better cross-modal knowledge sharing? Could the authors evaluate visual rendering on other existing multimodal retrieval architectures?

3.What is the end-to-end computational overhead? Does visual rendering substantially increase token counts, encoding latency, or training costs compared with native-input approaches?

4.Can the authors explain the performance degradation observed on certain visual grounding tasks?

5.MMEB has already been updated to V3, and MMEB-V2 introduced Visual Document Retrieval as a dedicated task category. Although the paper evaluates on ViDoRe v2, why do the authors not consider evaluating on newer MMEB versions? Experiments on the relevant subsets of MMEB-V2 or MMEB-V3 would provide stronger evidence of the proposed method's generalization and effectiveness.

10 comments
AuthorsOpenReview ↗

Response to Reviewer hh9j (Part 1/3)

We sincerely thank Reviewer hh9j for the detailed feedback and for recognizing our input-level unification perspective and the value of our ablation studies.

Q1/W1: Why should pixel-level unification improve over native multimodal inputs, and under what conditions is it beneficial?

With native multimodal inputs, images pass through a visual encoder and projector, while text passes through a text embedding layer. The resulting tokens have the same dimensionality but modality-dependent distributions, requiring mappings from different input representation spaces to a common retrieval space. Instead, pixel-level unification in RenderEmbed enables one mapping from visual inputs to retrieval embeddings, allowing heterogeneous supervision data to jointly optimize the same pathway.

The original experiments support this design:

  • Table 3: Under the same InfoNCE objective and 25% training subset, visual unification improves Overall Precision@1 by 0.6 points; the complete framework improves over NativeEmbed by 1.1 points.
  • Figure 3a: The complete framework's advantage reaches 1.5 points with full training data.
  • Table 1: Gains are larger on OOD tasks than on in-domain tasks, reaching 2.1 points over MetaEmbed and 4.4 points over MoCa-3B on OOD tasks.

Applicable conditions. Our results support visual unification in joint training across heterogeneous text, image, and composed-input tasks, particularly for transfer to held-out tasks or unseen input configurations. The transfer experiment in Q2/W2 further shows larger gains as heterogeneous supervision is added.


Q2/W2: Do the improvements reflect better cross-modal knowledge sharing, and does visual rendering benefit other retrieval architectures?

Thank you for suggesting these insightful evaluations, which strengthen our assessment of cross-modal transfer and generalization across architectures.

(1) Transfer across input configurations. We train NativeEmbed and RenderEmbed on either a target group TT or T+ST+S:

  • TT: MSCOCO_t2i and VisualNews_t2i, both text-to-image retrieval, with 25K examples each.
  • SS: CIRR, text-and-image-to-image retrieval, with 25K examples.
  • NIGHTS: image-to-image retrieval, excluded from both training settings.

Scores are Precision@1 (%). Under TT-only training, both CIRR and NIGHTS represent unseen input configurations.

Training dataModelMSCOCO_t2iVisualNews_t2iCIRRNIGHTSAvg.
TTNativeEmbed75.873.918.264.458.1
RenderEmbed76.173.219.165.358.4
Δ\Delta+0.3−0.7+0.9+0.9+0.3
T+ST+SNativeEmbed76.173.361.766.169.3
RenderEmbed76.373.763.167.670.2
Δ\Delta+0.2+0.4+1.4+1.5+0.9

Here, Δ\Delta denotes RenderEmbed minus NativeEmbed.

With TT alone, RenderEmbed has no consistent advantage on the two training datasets, yet outperforms NativeEmbed by 0.9 points on both CIRR and NIGHTS. This supports transfer from text-to-image supervision to composed-query and image-query retrieval, consistent with the OOD behavior in the paper.

Adding SS improves all four datasets for RenderEmbed, while NativeEmbed's VisualNews score decreases from 73.9 to 73.3. RenderEmbed's advantage also widens from 0.9 to 1.4 points on CIRR and from 0.9 to 1.5 points on NIGHTS. These trends suggest reduced interference between heterogeneous tasks and more effective use of additional supervision, consistent with Figure 3a.

(2) Transfer to a single-vector architecture. We apply Visual Substrate Rendering (VSR) & Interventional Rendering Learning (IRL) to the official single-vector VLM2Vec architecture, without our Progressive Hierarchical Anchor Encoding. We compare against native inputs using Qwen2.5-VL-3B and the same 25% training subset as Table 3. Results are Precision@1 (%).

VLM2Vec input / training settingClassificationVQARetrievalGroundingINDOODOverall
Native inputs59.1761.5866.4480.9869.0159.2864.69
VSR & IRL62.5765.3167.4583.7870.3163.5767.31

All four task categories improve, with a larger gain on OOD tasks (4.29 points) than on in-domain tasks (1.30 points). These gains show that visual unification with its accompanying intervention objectives also benefits a single-vector retriever.

AuthorsOpenReview ↗

Response to Reviewer hh9j (Part 2/3)

Q3/W3: How does visual rendering affect token counts, encoding latency, and training costs?

Together with the retrieval-stage measurements in Table 4, the following training and encoding results provide a breakdown of costs across the main stages.

For training and encoding, we compare with NativeEmbed, which shares the backbone and anchor/fusion architecture, to measure the additional costs of visual rendering and its training objectives. These measurements use a single NVIDIA H20 with BF16 and five repetitions. Inputs are sampled from 26 subsets, totaling 1,664 query–candidate pairs.

(1) Training cost. Throughput is measured at batch size 4 in query–candidate pairs per second, counting each pair once even when an alternative rendering view is added.

Training settingThroughput (pairs/s)
NativeEmbed2.381 ± 0.003
RenderEmbed without alternative rendering view1.981 ± 0.005
RenderEmbed with alternative rendering view1.547 ± 0.002

Rendering and alternative-view encoding increase training time by approximately 53.9% for the same number of original training pairs relative to NativeEmbed. Alternative views are used only during training, without inference-time costs.

(2) Query and candidate encoding. We measure latency from raw input to embedding, including preprocessing and rendering, at batch size 1.

MethodAvg. query tokensAvg. candidate tokensQuery latency (ms/query)Candidate latency (ms/candidate)
NativeEmbed385.5167.6136.97 ± 1.7796.25 ± 0.40
RenderEmbed454.2281.7166.31 ± 0.41134.90 ± 0.61

Rendering itself takes 25.33 ms/query and 13.73 ms/candidate, already included above. Encoding text as pixels also introduces visual processing and longer backbone input sequences: query and candidate token counts increase by 17.8% and 68.1%, respectively. The additional online query-encoding latency is 29.34 ms (+21.4%). Candidate encoding is performed offline during index construction or updates.


Q4/W4: What causes the performance degradation on certain visual grounding tasks?

We thank the reviewer for suggesting this valuable analysis, helping us improve our work. We examine the four visual grounding subsets of MMEB: MSCOCO, Visual7W-Pointing, RefCOCO, and RefCOCO-Matching, using NativeEmbed with the same backbone as a control. Results are Precision@1 (%).

MethodMSCOCOVisual7W-PointingRefCOCORefCOCO-MatchingAvg.
NativeEmbed74.585.386.886.183.2
RenderEmbed67.985.192.587.483.2
RenderEmbed + adaptive enlargement72.087.994.087.485.3

RenderEmbed's main deficit relative to NativeEmbed occurs in the MSCOCO grounding subset (67.9 vs. 74.5).

(1) Insufficient resolution for small objects. The MSCOCO grounding subset requires retrieving an object crop from a full scene image and a target-category instruction. For 33.1% of queries, the positive candidate crop is smaller than 4,096 pixels, and RenderEmbed achieves only 32.6% Precision@1 on this group.

Our renderer places these crops alongside instruction text without enlarging them. Text and whitespace can make the canvas satisfy the processor's minimum pixel budget while the object itself remains small. NativeEmbed resizes each candidate image independently, allowing small crops to reach the minimum image resolution. This difference suggests that insufficient local object resolution contributes to the degradation.

(2) Adaptive enlargement before rendering. Following the native image processor's resizing rule, we enlarge undersized candidate images before composing the visual canvas. Applied at inference without retraining, this adjustment improves MSCOCO by 4.1 points and the grounding average by 2.1 points, alleviating the observed degradation.

We plan to apply the same resizing policy to the training data, which we expect to further improve grounding performance.

AuthorsOpenReview ↗

Response to Reviewer hh9j (Part 3/3)

Q5: Why were newer MMEB versions not evaluated, and can the authors provide results on MMEB-V2 or MMEB-V3?

(1) Benchmark selection for baseline comparison. We selected MMEB-V1 and ViDoRe v2 to facilitate direct comparison with existing baselines. Most methods in Table 1 were evaluated under these established benchmark settings, without corresponding MMEB-V2 or MMEB-V3 results reported in their original papers.

(2) Additional evaluation on MMEB-V2. Following the reviewer's suggestion, we evaluate all 42 tasks added in MMEB-V2, covering 18 video and 24 visual-document tasks. We use the same model weights as in Table 1, without additional video training. Officially sampled video keyframes are rendered together into a single visual input. We compare with published results in VLM2Vec-V2 [2]; “Best reported” denotes the column-wise maximum across models in [2].

Video results (Precision@1, %):

MethodCLSQARETMRETOverall (18 tasks)
VLM2Vec-V2 [2]39.334.328.838.534.9
Best reported in [2]39.350.429.040.638.6
RenderEmbed45.7354.5643.3145.6247.49

Visual-document results (NDCG@5, %):

MethodVDRv1VDRv2VRVisDoc OODOverall (24 tasks)
VLM2Vec-V2 [2]75.544.979.439.465.4
Best reported in [2]89.455.685.044.475.2
RenderEmbed90.1959.8488.5770.5381.45

RenderEmbed exceeds the best reported overall values in [2] by 8.89 points on video and 6.25 points on visual documents, extending the original evaluation to a broader set of tasks.

(3) Audio evaluation in MMEB-V3. MMEB-V3 further introduces audio tasks. Raw audio generally requires an audio-specific encoder, which neither our framework nor the evaluated baselines provide. In future work, we plan to explore rendering speech transcripts and spectrograms as visual inputs for a common visual interface.

[2] Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. TMLR, 2026.


W4: Do progressively interleaved anchors learn useful hierarchical representations?

(1) Meaning of hierarchical and progressive. In our method, hierarchical refers to cross-layer fusion of visual representations at different depths and granularities. Progressive interleaving determines the expanding visual contexts accessible to successive anchors. Table 3 shows a 0.7-point drop when using only final-layer states, and the fusion-layer analysis in Figure 4 finds that all evaluated cross-layer configurations outperform the final-layer-only baseline.

(2) Retrieval evidence across anchor positions. We fix all 16 query anchors and vary only the retained candidate anchors, using the original MaxSim scoring. RenderEmbed and the appended-anchor variant have matched training settings.

Numbered columns retain a single candidate anchor at that position. Tail uses {13,14,15,16}, Spread uses {4,8,12,16}, and Full uses all 16. Results are reported in Precision@1 (%).

Model1481216TailSpreadFull
w Appended Anchors64.3164.0664.8365.0460.8367.267.768.2
RenderEmbed34.9657.6862.7864.6965.8966.468.068.6

Position-dependent behavior. Appended anchors yield relatively similar single-anchor scores across the sampled positions, with no progressive increase. RenderEmbed shows a clear increase from 34.96 to 65.89, consistent with successive anchors accessing broader visual contexts.

Complementarity across positions. For RenderEmbed, Spread scores 68.0 versus 66.4 for Tail at the same four-vector budget; the 1.6-point gain supports complementarity across different context ranges.

Reviewer_fedJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed unification before encoding paradigm that convert multimodal inputs into visual tokens with visual substrate rendering. It uses progressive hierarchical anchor encoding to retain retrieval representation of different levels, and interventional rendering learning to distinguish retrieval semantics from presentation variations. Experiments on MMEB and ViDoRe v2 demenstrate notable performance and out-of domain generalization of RenderEmbed.

Critical strengths

1.Insteresting research idea: the paper propose an interesting research idea of coverting multimodal inputs into visual tokens, unlike traditional vision-language alignment methods.

2.Systematic methods and Notable results: the proposed progressive hierarchical anchor encoding and interventional rendering learning is well-designed to adapt to the task. the experiment results are relatively notable, showing good performance on retrieval and other out-of-domain motimodal tasks.

Critical weaknesses

1.Limited novelty and contribution: the idea of converting inputs to visual tokens has already been explored in previous works, and the progressive hierarchical anchor encoding seems to be a small improvement of previous methods. The ablation study (Table 3) does not provide enough performance gain proving the effectiveness of these proposed methods (only 0.5 performance gain with interventional rendering learning).

2.Unclear idea: the paper does not discuss why converting inputs to visual tokens is better than traditional vision-language alignment methods (or the necessity of using this method).

Questions for authors

1.What is the difference between RenderEmbed and previous methods that converting texts to images? Is it really necessary to convert inputs to visual tokens?

2.Does RenderEmbed work on other multimodal tasks like generation, or tasks with audio or video as inputs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fedJ (Part 1/2)

We sincerely thank Reviewer fedJ for the thoughtful feedback and for recognizing our input-unification idea, systematic method design, and strong retrieval and out-of-domain performance.

Q1/W1/W2: How does RenderEmbed differ from previous methods that convert text to images, and why convert inputs to visual tokens?

(1) Difference from prior visual encoding methods. CLIPPO learns pixel-based representations from image–text and text–text contrastive pairs, followed by task-specific adaptation; its retrieval evaluation focuses on image–text matching. UniSE encodes prepared screenshots as visual tokens while retaining text tokens for textual queries and candidates, without using a unified visual input format for multimodal retrieval. Its evaluation centers on screenshot-based tasks.

RenderEmbed renders text, images, and their compositions on both retrieval sides into a common visual substrate. A single embedding model supports classification, VQA, retrieval, and visual grounding (Tables 1–2).

(2) Motivation and evidence in the paper. With native inputs, images pass through a visual encoder and projector, while text passes through a text embedding layer. The resulting tokens have modality-dependent distributions, requiring mappings from different input representations and their combinations to a common retrieval space. By unifying input format, RenderEmbed instead learns a single mapping from visual inputs to retrieval embeddings, allowing heterogeneous supervision data to jointly optimize the same pathway.

The paper provides supporting evidence:

  • Table 3 and Figure 3a: Visual rendering alone improves over NativeEmbed by 0.6 points on the 25% training subset. The complete framework improves by 1.1 points on this subset and 1.5 points with full training data.
  • Table 1 reports larger gains on OOD tasks than on in-domain tasks, including OOD gains of 2.1 points over MetaEmbed and 4.4 points over MoCa-3B.

(3) Additional transfer evidence. Training only on MSCOCO_t2i and VisualNews_t2i (text-to-image, 25K examples each), RenderEmbed has no consistent advantage on these two tasks, yet improves over NativeEmbed by 0.9 points on both unseen configurations: CIRR (text-and-image-to-image) and NIGHTS (image-to-image). Adding 25K CIRR examples improves all four datasets for RenderEmbed, whereas NativeEmbed's VisualNews score decreases. RenderEmbed's advantage also widens to 1.4 points on CIRR and 1.5 points on NIGHTS, which remains excluded from training.

These results support transfer to unseen input configurations and suggest reduced interference during joint training.


Q2: Does RenderEmbed support generation or tasks with audio or video inputs?

(1) Generation and audio tasks. Our study focuses on multimodal embedding and retrieval. Thus, generation tasks fall outside this scope. Audio inputs generally require an audio-specific encoder, which our framework and the evaluated baselines do not include. In future work, we plan to explore extending the visual interface to audio by rendering speech transcripts and spectrograms as visual inputs.

(2) Video and additional MMEB-V2 tasks. We evaluate all 42 tasks added in MMEB-V2, comprising 18 video and 24 visual-document tasks. We use the same model weights as in Table 1, without additional video training. For videos, we use the keyframes sampled by the official evaluation code and render them together into one image through Visual Substrate Rendering.

Since some Table 1 baselines have no reported MMEB-V2 results, we compare with published results in VLM2Vec-V2 [2]. “Best reported” denotes the column-wise maximum across models in [2].

Video results (Precision@1, %)

MethodCLSQARETMRETOverall (18 tasks)
VLM2Vec-V2 [2]39.334.328.838.534.9
Best reported in [2]39.350.429.040.638.6
RenderEmbed45.7354.5643.3145.6247.49

Visual-document results (NDCG@5, %)

MethodVDRv1VDRv2VRVisDoc OODOverall (24 tasks)
VLM2Vec-V2 [2]75.544.979.439.465.4
Best reported in [2]89.455.685.044.475.2
RenderEmbed90.1959.8488.5770.5381.45

RenderEmbed exceeds the best reported overall scores in [2] by 8.89 points on video and 6.25 points on visual documents. The video results demonstrate that the shared visual interface accommodates sampled video frames without video-specific training.

[2] Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. TMLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer fedJ (Part 2/2)

W1: How do the ablations support the anchor design and interventional learning, given the 0.5-point gain in Table 3?

(1) Component contributions to retrieval accuracy. Table 3 evaluates all variants using 25% of the training data. Replacing progressive interleaving with appended anchors, using only final-layer states, and removing Interventional Rendering Learning reduce Overall Precision@1 by 0.4, 0.7, and 0.5 points, respectively. These ablations support the contributions of context aggregation, cross-layer fusion, and interventional learning to retrieval accuracy.

(2) Rendering robustness. Interventional learning also targets stability across presentation changes. Figure 3c shows that, compared with removing the rendering-consistency objective, the full model exhibits more favorable changes relative to canonical rendering under 8 of 10 alternative policies, with smaller mean degradation and a less severe worst-case drop. This provides evidence of robustness beyond the canonical-rendering accuracy measured in Table 3.

(3) Framework performance with more training data. At the framework level, RenderEmbed’s advantage over NativeEmbed increases from 1.1 points in the 25%-data ablation to 1.5 points with full training data (Figure 3a), supporting its effectiveness at a larger training scale.

Reviewer_NCk2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RenderEmbed, a unified multimodal embedding framework that converts text, images, and their combinations into a common visual representation before encoding. The method introduces progressively interleaved anchor tokens with cross-layer fusion to produce compact multi-vector embeddings, and employs semantic and rendering interventions to improve retrieval accuracy and robustness. Experiments on MMEB and ViDoRe v2 demonstrate improvements over several existing multimodal embedding models.

Critical strengths
  1. The paper is clearly organized, and the proposed framework provides a coherent pipeline for unified multimodal retrieval.
  2. The experiments cover diverse multimodal retrieval tasks, with improvements on MMEB and competitive results on ViDoRe v2. The authors also provide ablation studies and retrieval efficiency analyses.
Critical weaknesses
  1. I think the main concern of this work is the proposed anchor placement, cross-layer fusion, and intervention losses are relatively incremental modifications. Rendering text as images has been explored in CLIPPO and UniSE, while the multi-vector architecture largely follows MetaEmbed.
  2. The paper argues that a shared visual encoding pathway improves cross-modal knowledge transfer, but provides no direct evidence of this. Visual rendering alone brings only a 0.6-point improvement under standard InfoNCE training.
  3. Replacing entire text or image elements does not necessarily produce fine-grained semantic negatives, and the validity of these constructed samples is not examined.
Questions for authors
  1. Can the authors provide direct evidence that unified visual encoding improves cross-modal knowledge transfer?
  2. How does RenderEmbed compare with MetaEmbed under the same vector budget, including end-to-end encoding costs?
  3. How do the proposed intervention objectives compare with standard hard-negative mining and rendering augmentation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NCk2 (Part 1/3)

We sincerely thank Reviewer NCk2 for the constructive feedback and for recognizing the coherence of our framework and the breadth of our multimodal evaluation.

Q1/W2: Can the authors provide direct evidence that unified visual encoding improves cross-modal knowledge transfer?

Thank you for requesting this valuable comment, which strengthens our evaluation of cross-modal knowledge transfer.

To provide direct evidence, we compare NativeEmbed and RenderEmbed under two training settings: target data only (TT) and target plus auxiliary data (T+ST+S). The training and evaluation groups are:

  • Target group TT: MSCOCO_t2i and VisualNews_t2i, both text-to-image retrieval, with 25K examples each.
  • Auxiliary group SS: CIRR, text-and-image-to-image retrieval, with 25K examples.
  • OOD evaluation: NIGHTS, image-to-image retrieval, excluded from both training settings.
Training dataModelMSCOCO_t2iVisualNews_t2iCIRRNIGHTSAvg.
TTNativeEmbed75.873.918.264.458.1
RenderEmbed76.173.219.165.358.4
Δ\Delta+0.3−0.7+0.9+0.9+0.3
T+ST+SNativeEmbed76.173.361.766.169.3
RenderEmbed76.373.763.167.670.2
Δ\Delta+0.2+0.4+1.4+1.5+0.9

Under TT-only training, neither CIRR nor NIGHTS receives training supervision. Δ\Delta denotes RenderEmbed minus NativeEmbed. The results reveal three patterns:

  • Transfer to unseen input configurations. With only text-to-image training, RenderEmbed shows no consistent advantage on the two target datasets, yet outperforms NativeEmbed by 0.9 points on both CIRR and NIGHTS. This supports transfer from text-to-image supervision to composed-query and image-query retrieval, consistent with the OOD results in Table 1.
  • Joint training across heterogeneous tasks. Adding CIRR decreases NativeEmbed’s VisualNews score from 73.9 to 73.3, while RenderEmbed improves from 73.2 to 73.7 and also improves on all other evaluated datasets. NativeEmbed retains modality-dependent input routes, whereas RenderEmbed processes all configurations through the same visual pathway. These trends suggest reduced interference between heterogeneous tasks through input unification.
  • Growing benefits from heterogeneous supervision. After adding SS, RenderEmbed’s advantage widens from 0.9 to 1.4 on CIRR and from 0.9 to 1.5 on NIGHTS. The widening gains suggest more effective use of heterogeneous supervision, consistent with the Training-Data Scaling results (Figure 3a).
AuthorsOpenReview ↗

Response to Reviewer NCk2 (Part 2/3)

Q2: How does RenderEmbed compare with MetaEmbed under the same vector budget, including end-to-end encoding costs?

(1) Retrieval accuracy at matched vector budgets. RenderEmbed outperforms MetaEmbed-3B at all four shared budgets, with gains of 0.3–3.2 points in MMEB Overall Precision@1. MetaEmbed results are taken from Table 5 of the original paper [1], and RenderEmbed results are reported in Figure 8(a) of our appendix.

Query / Candidate vectorsMetaEmbed-3BRenderEmbedΔ\Delta
(1, 1)65.465.7+0.3
(2, 4)65.868.3+2.5
(4, 8)66.769.8+3.1
(8, 16)67.570.7+3.2

(2) End-to-end encoding costs. We measure encoding latency from raw input to embedding, including preprocessing and rendering. We compare with NativeEmbed, which uses the same backbone and anchor/fusion architecture with native multimodal inputs, to quantify the additional cost introduced by visual rendering.

Measurements use a single NVIDIA H20 (BF16, batch size 1) on 1,664 query–candidate pairs from 26 subsets. Token counts are averaged, and encoding latency is reported over five runs as mean ± standard deviation.

MethodQuery tokensCandidate tokensQuery latency (ms)Candidate latency (ms)
NativeEmbed385.5167.6136.97 ± 1.7796.25 ± 0.40
RenderEmbed454.2281.7166.31 ± 0.41134.90 ± 0.61
  • Rendering takes 25.33 ms/query and 13.73 ms/candidate, already included in the reported latency. Encoding text as pixels also requires visual processing and lengthens backbone input sequences, increasing query and candidate token counts by 17.8% and 68.1%, respectively.
  • Online query-encoding latency increases by 29.34 ms (+21.4%). The 38.65 ms (+40.2%) increase in candidate-encoding latency affects offline index construction and updates.

[1] Xiao et al. MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. ICLR 2026.


Q3: How do the proposed intervention objectives compare with standard hard-negative mining and rendering augmentation?

We compare the two interventions separately using the same 25% training subset and experimental setup as Table 3.

(1) Comparison with standard hard-negative mining. In Table 3, w/o Semantic Intervention replaces the rendering-controlled negatives with standard mined hard negatives, while retaining the rendering-consistency objective. RenderEmbed improves Overall Precision@1 from 68.1 to 68.6 (+0.5 points).

(2) Comparison with ordinary rendering augmentation. We add a baseline that replaces the margin-consistency loss with direct contrastive retrieval supervision on the alternative rendering view. Specifically, Lalt-ret\mathcal{L}_{\mathrm{alt\text{-}ret}} follows Eq.(7), replacing the positive score siis_{ii} with s~ii\widetilde{s}_{ii} and the in-batch negative scores sijs_{ij} with s~ij\widetilde{s}_{ij} (j≠ij\neq i), while removing semantic-negative score s^ii\hat{s}_{ii}. Here, s~ij=s(ψ∗(qi),ψ′(dj+))\widetilde{s}_{ij}=s\bigl(\psi^*(q_i),\psi'(d_j^+)\bigr) pairs the canonical query with the alternatively rendered candidate. The total loss is L=Lret+0.1,Lalt-ret\mathcal{L}=\mathcal{L}_{\mathrm{ret}}+0.1\\,\mathcal{L}_{\mathrm{alt\text{-}ret}}, retaining the original-view retrieval objective.

ΔP@1 denotes the change in Overall Precision@1 relative to canonical rendering, reported as mean ± standard deviation across 10 rendering policies in percentage points (pp). Negative values indicate degradation.

MethodOverall Precision@1 (%)ΔP@1 (pp)
RenderEmbed68.6−0.12 ± 0.09
w/o Rendering Intervention68.4−0.27 ± 0.13
w Rendering Augmentation67.9−0.12 ± 0.11

Both rendering augmentation and our consistency objective reduce the average degradation relative to removing the rendering intervention. Our objective achieves 0.7 points higher retrieval accuracy than ordinary augmentation (68.6 vs. 67.9), with the same mean change of −0.12 points. Our consistency objective explicitly aligns positive-to-negative score margins across rendering views, encouraging stable retrieval decisions under presentation changes.

AuthorsOpenReview ↗

Response to Reviewer NCk2 (Part 3/3)

W1: What is the novelty of RenderEmbed compared with CLIPPO, UniSE, and MetaEmbed?

As acknowledged in Related Work, visual rendering and multi-vector architectures have precedents. Detailed distinctions and contributions are as follows.

(1) Difference from CLIPPO and UniSE. CLIPPO learns pixel-based representations from image–text and text–text contrastive pairs, followed by task-specific adaptation; its retrieval evaluation focuses on image–text matching. UniSE encodes prepared screenshots as visual tokens while retaining text tokens for textual queries and candidates, without using a unified visual input format for multimodal retrieval. Its evaluation centers on screenshot-based tasks.

RenderEmbed renders text, images, and their compositions on both retrieval sides into a common visual substrate, enabling a shared visual encoding pathway across heterogeneous multimodal tasks. A single embedding model supports classification, VQA, retrieval, and visual grounding (Tables 1–2).

(2) Difference from MetaEmbed. We follow MetaEmbed’s compact multi-vector architecture and adapt it to unified visual inputs. Progressive interleaving lets successive anchors aggregate expanding visual contexts, and cross-layer fusion combines multi-granularity visual representations from different depths. Table 3 supports both choices.

The proposed anchor encoding further supports test-time budget adaptation without the Matryoshka Multi-Vector Retrieval objective of MetaEmbed (see Section 4.4).

(3) Substantive contribution. Our contribution is a unified multimodal retrieval framework built on unification-before-encoding paradigm. A common visual substrate requires encoding distributed evidence at multiple granularities while separating content semantics from presentation variations. Progressive Hierarchical Anchor Encoding and Interventional Rendering Learning address these complementary requirements.

Together with Visual Substrate Rendering, they enable heterogeneous task supervision to train one visual encoding pathway with compact, content-sensitive, and rendering-robust representations.


W2: Visual rendering alone brings only a 0.6-point improvement under standard InfoNCE.

Visual rendering and interventional learning are complementary. Rendering can introduce presentation variations that encourage shortcuts tied to appearance. Interventional Rendering Learning addresses this through complementary content and rendering interventions that distinguish retrieval semantics from presentation variations. These interventions reuse existing training pairs and require no additional source data or annotations.

On the same 25% training subset, rendering alone improves Overall Precision@1 by 0.6 points, while the complete framework improves it by 1.1 points (Table 3). The advantage over NativeEmbed reaches 1.5 points with the full training data (Figure 3a).


W3: How does replacing entire text or image elements produce fine-grained semantic negatives, and are these negatives valid?

(1) Fine-grained semantic differences in composed candidates. For composed image–text retrieval, relevance requires a correctly matched image–text pair. Existing negatives often consist of internally matched pairs that are simply irrelevant to the query. Our intervention replaces one element of a positive candidate with its counterpart from a negative, creating a partially correct but mismatched pair. Preserving one correct component while altering the other encourages learning of fine-grained semantic distinctions within composed candidates.

(2) Negative validity and manual evaluation. Under this relevance criterion, a partially correct but mismatched pair remains a negative. Additionally, if the constructed candidate is identical to the positive, our algorithm falls back to the original negative.

We sampled 20 examples from each of 26 training subsets, yielding 520 examples. Two annotators assessed negative validity, reporting rates of 99% and 98.8%, respectively. We inspected the union of their flagged cases, comprising six examples:

  • Three from tevatron_colpali: the original positive and negative referenced identical images under different filenames.
  • Three from MSCOCO_i2t: the original negative captions closely matched the positive descriptions and were judged false negatives by the annotators.

All six observed false negatives were inherited from the source annotations. We plan to investigate methods for detecting and filtering these annotation issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.