- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RenderEmbed, a unified multimodal embedding framework based on Unification before Encoding. The method converts text, images, and composed multimodal inputs into a common visual substrate, and employs Progressive Hierarchical Anchor Encoding and Interventional Rendering Learning to produce compact and robust multi-vector representations. Experiments on MMEB and ViDoRe v2 demonstrate competitive retrieval performance.
Critical strengths
1.Interesting research perspective. Unlike existing approaches that primarily focus on shared encoders or feature-space alignment, this paper explores unification at the input level, offering an interesting direction for unified multimodal representation learning.
2.The paper provides ablation studies on major components, anchor budgets, cross-layer fusion, and rendering robustness, helping to assess the effectiveness of different design choices.
Critical weaknesses
1.Unifying input formats does not necessarily lead to better cross-modal semantic alignment. Existing VLMs already integrate textual and visual information through shared backbones. The paper does not sufficiently explain why converting native text tokens into visual inputs should systematically improve representation learning, or whether this design generalizes to other models and tasks.
2.Although the complete RenderEmbed framework improves over NativeEmbed by 1.1 percentage points, the independent gain from visual input unification under standard InfoNCE is only 0.6 percentage points. The current experiments do not adequately demonstrate that the improvements arise from better knowledge sharing across heterogeneous modalities. Additional cross-modal transfer experiments and evaluations of pixel rendering on other multimodal retrieval architectures would strengthen this claim.
3.Rendering text into images may increase visual token counts and encoding overhead. However, the efficiency analysis mainly focuses on index storage and retrieval scoring, without fully accounting for rendering and encoding costs. Additional measurements of encoding latency, training throughput, and end-to-end retrieval latency are needed.
4.it remains unclear whether progressively interleaved anchors learn meaningful hierarchical semantic representations. Furthermore, the model exhibits noticeable performance degradation on certain visual grounding tasks, which warrants further investigation.
Questions for authors
1.Why should pixel-level unification outperform native multimodal inputs? Can the authors provide a clearer explanation or empirical evidence demonstrating its unique benefits and applicable conditions?
2.Do the performance improvements genuinely originate from better cross-modal knowledge sharing? Could the authors evaluate visual rendering on other existing multimodal retrieval architectures?
3.What is the end-to-end computational overhead? Does visual rendering substantially increase token counts, encoding latency, or training costs compared with native-input approaches?
4.Can the authors explain the performance degradation observed on certain visual grounding tasks?
5.MMEB has already been updated to V3, and MMEB-V2 introduced Visual Document Retrieval as a dedicated task category. Although the paper evaluates on ViDoRe v2, why do the authors not consider evaluating on newer MMEB versions? Experiments on the relevant subsets of MMEB-V2 or MMEB-V3 would provide stronger evidence of the proposed method's generalization and effectiveness.
10 comments
Response to Reviewer hh9j (Part 2/3)
Q3/W3: How does visual rendering affect token counts, encoding latency, and training costs?
Together with the retrieval-stage measurements in Table 4, the following training and encoding results provide a breakdown of costs across the main stages.
For training and encoding, we compare with NativeEmbed, which shares the backbone and anchor/fusion architecture, to measure the additional costs of visual rendering and its training objectives. These measurements use a single NVIDIA H20 with BF16 and five repetitions. Inputs are sampled from 26 subsets, totaling 1,664 query–candidate pairs.
(1) Training cost. Throughput is measured at batch size 4 in query–candidate pairs per second, counting each pair once even when an alternative rendering view is added.
| Training setting | Throughput (pairs/s) |
|---|---|
| NativeEmbed | 2.381 ± 0.003 |
| RenderEmbed without alternative rendering view | 1.981 ± 0.005 |
| RenderEmbed with alternative rendering view | 1.547 ± 0.002 |
Rendering and alternative-view encoding increase training time by approximately 53.9% for the same number of original training pairs relative to NativeEmbed. Alternative views are used only during training, without inference-time costs.
(2) Query and candidate encoding. We measure latency from raw input to embedding, including preprocessing and rendering, at batch size 1.
| Method | Avg. query tokens | Avg. candidate tokens | Query latency (ms/query) | Candidate latency (ms/candidate) |
|---|---|---|---|---|
| NativeEmbed | 385.5 | 167.6 | 136.97 ± 1.77 | 96.25 ± 0.40 |
| RenderEmbed | 454.2 | 281.7 | 166.31 ± 0.41 | 134.90 ± 0.61 |
Rendering itself takes 25.33 ms/query and 13.73 ms/candidate, already included above. Encoding text as pixels also introduces visual processing and longer backbone input sequences: query and candidate token counts increase by 17.8% and 68.1%, respectively. The additional online query-encoding latency is 29.34 ms (+21.4%). Candidate encoding is performed offline during index construction or updates.
Q4/W4: What causes the performance degradation on certain visual grounding tasks?
We thank the reviewer for suggesting this valuable analysis, helping us improve our work. We examine the four visual grounding subsets of MMEB: MSCOCO, Visual7W-Pointing, RefCOCO, and RefCOCO-Matching, using NativeEmbed with the same backbone as a control. Results are Precision@1 (%).
| Method | MSCOCO | Visual7W-Pointing | RefCOCO | RefCOCO-Matching | Avg. |
|---|---|---|---|---|---|
| NativeEmbed | 74.5 | 85.3 | 86.8 | 86.1 | 83.2 |
| RenderEmbed | 67.9 | 85.1 | 92.5 | 87.4 | 83.2 |
| RenderEmbed + adaptive enlargement | 72.0 | 87.9 | 94.0 | 87.4 | 85.3 |
RenderEmbed's main deficit relative to NativeEmbed occurs in the MSCOCO grounding subset (67.9 vs. 74.5).
(1) Insufficient resolution for small objects. The MSCOCO grounding subset requires retrieving an object crop from a full scene image and a target-category instruction. For 33.1% of queries, the positive candidate crop is smaller than 4,096 pixels, and RenderEmbed achieves only 32.6% Precision@1 on this group.
Our renderer places these crops alongside instruction text without enlarging them. Text and whitespace can make the canvas satisfy the processor's minimum pixel budget while the object itself remains small. NativeEmbed resizes each candidate image independently, allowing small crops to reach the minimum image resolution. This difference suggests that insufficient local object resolution contributes to the degradation.
(2) Adaptive enlargement before rendering. Following the native image processor's resizing rule, we enlarge undersized candidate images before composing the visual canvas. Applied at inference without retraining, this adjustment improves MSCOCO by 4.1 points and the grounding average by 2.1 points, alleviating the observed degradation.
We plan to apply the same resizing policy to the training data, which we expect to further improve grounding performance.
Response to Reviewer hh9j (Part 3/3)
Q5: Why were newer MMEB versions not evaluated, and can the authors provide results on MMEB-V2 or MMEB-V3?
(1) Benchmark selection for baseline comparison. We selected MMEB-V1 and ViDoRe v2 to facilitate direct comparison with existing baselines. Most methods in Table 1 were evaluated under these established benchmark settings, without corresponding MMEB-V2 or MMEB-V3 results reported in their original papers.
(2) Additional evaluation on MMEB-V2. Following the reviewer's suggestion, we evaluate all 42 tasks added in MMEB-V2, covering 18 video and 24 visual-document tasks. We use the same model weights as in Table 1, without additional video training. Officially sampled video keyframes are rendered together into a single visual input. We compare with published results in VLM2Vec-V2 [2]; “Best reported” denotes the column-wise maximum across models in [2].
Video results (Precision@1, %):
| Method | CLS | QA | RET | MRET | Overall (18 tasks) |
|---|---|---|---|---|---|
| VLM2Vec-V2 [2] | 39.3 | 34.3 | 28.8 | 38.5 | 34.9 |
| Best reported in [2] | 39.3 | 50.4 | 29.0 | 40.6 | 38.6 |
| RenderEmbed | 45.73 | 54.56 | 43.31 | 45.62 | 47.49 |
Visual-document results (NDCG@5, %):
| Method | VDRv1 | VDRv2 | VR | VisDoc OOD | Overall (24 tasks) |
|---|---|---|---|---|---|
| VLM2Vec-V2 [2] | 75.5 | 44.9 | 79.4 | 39.4 | 65.4 |
| Best reported in [2] | 89.4 | 55.6 | 85.0 | 44.4 | 75.2 |
| RenderEmbed | 90.19 | 59.84 | 88.57 | 70.53 | 81.45 |
RenderEmbed exceeds the best reported overall values in [2] by 8.89 points on video and 6.25 points on visual documents, extending the original evaluation to a broader set of tasks.
(3) Audio evaluation in MMEB-V3. MMEB-V3 further introduces audio tasks. Raw audio generally requires an audio-specific encoder, which neither our framework nor the evaluated baselines provide. In future work, we plan to explore rendering speech transcripts and spectrograms as visual inputs for a common visual interface.
[2] Meng et al. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. TMLR, 2026.
W4: Do progressively interleaved anchors learn useful hierarchical representations?
(1) Meaning of hierarchical and progressive. In our method, hierarchical refers to cross-layer fusion of visual representations at different depths and granularities. Progressive interleaving determines the expanding visual contexts accessible to successive anchors. Table 3 shows a 0.7-point drop when using only final-layer states, and the fusion-layer analysis in Figure 4 finds that all evaluated cross-layer configurations outperform the final-layer-only baseline.
(2) Retrieval evidence across anchor positions. We fix all 16 query anchors and vary only the retained candidate anchors, using the original MaxSim scoring. RenderEmbed and the appended-anchor variant have matched training settings.
Numbered columns retain a single candidate anchor at that position. Tail uses {13,14,15,16}, Spread uses {4,8,12,16}, and Full uses all 16. Results are reported in Precision@1 (%).
| Model | 1 | 4 | 8 | 12 | 16 | Tail | Spread | Full |
|---|---|---|---|---|---|---|---|---|
| w Appended Anchors | 64.31 | 64.06 | 64.83 | 65.04 | 60.83 | 67.2 | 67.7 | 68.2 |
| RenderEmbed | 34.96 | 57.68 | 62.78 | 64.69 | 65.89 | 66.4 | 68.0 | 68.6 |
Position-dependent behavior. Appended anchors yield relatively similar single-anchor scores across the sampled positions, with no progressive increase. RenderEmbed shows a clear increase from 34.96 to 65.89, consistent with successive anchors accessing broader visual contexts.
Complementarity across positions. For RenderEmbed, Spread scores 68.0 versus 66.4 for Tail at the same four-vector budget; the 1.6-point gain supports complementarity across different context ranges.
Response to Reviewer hh9j (Part 1/3)
We sincerely thank Reviewer hh9j for the detailed feedback and for recognizing our input-level unification perspective and the value of our ablation studies.
Q1/W1: Why should pixel-level unification improve over native multimodal inputs, and under what conditions is it beneficial?
With native multimodal inputs, images pass through a visual encoder and projector, while text passes through a text embedding layer. The resulting tokens have the same dimensionality but modality-dependent distributions, requiring mappings from different input representation spaces to a common retrieval space. Instead, pixel-level unification in RenderEmbed enables one mapping from visual inputs to retrieval embeddings, allowing heterogeneous supervision data to jointly optimize the same pathway.
The original experiments support this design:
Applicable conditions. Our results support visual unification in joint training across heterogeneous text, image, and composed-input tasks, particularly for transfer to held-out tasks or unseen input configurations. The transfer experiment in Q2/W2 further shows larger gains as heterogeneous supervision is added.
Q2/W2: Do the improvements reflect better cross-modal knowledge sharing, and does visual rendering benefit other retrieval architectures?
Thank you for suggesting these insightful evaluations, which strengthen our assessment of cross-modal transfer and generalization across architectures.
(1) Transfer across input configurations. We train NativeEmbed and RenderEmbed on either a target group T or T+S:
text-to-image retrieval, with 25K examples each.text-and-image-to-image retrieval, with 25K examples.image-to-image retrieval, excluded from both training settings.Scores are Precision@1 (%). Under T-only training, both CIRR and NIGHTS represent unseen input configurations.
Here, Δ denotes RenderEmbed minus NativeEmbed.
With T alone, RenderEmbed has no consistent advantage on the two training datasets, yet outperforms NativeEmbed by 0.9 points on both CIRR and NIGHTS. This supports transfer from text-to-image supervision to composed-query and image-query retrieval, consistent with the OOD behavior in the paper.
Adding S improves all four datasets for RenderEmbed, while NativeEmbed's VisualNews score decreases from 73.9 to 73.3. RenderEmbed's advantage also widens from 0.9 to 1.4 points on CIRR and from 0.9 to 1.5 points on NIGHTS. These trends suggest reduced interference between heterogeneous tasks and more effective use of additional supervision, consistent with Figure 3a.
(2) Transfer to a single-vector architecture. We apply Visual Substrate Rendering (VSR) & Interventional Rendering Learning (IRL) to the official single-vector VLM2Vec architecture, without our Progressive Hierarchical Anchor Encoding. We compare against native inputs using Qwen2.5-VL-3B and the same 25% training subset as Table 3. Results are Precision@1 (%).
All four task categories improve, with a larger gain on OOD tasks (4.29 points) than on in-domain tasks (1.30 points). These gains show that visual unification with its accompanying intervention objectives also benefits a single-vector retriever.