acceptodds
Under review as a conference paper at ICLR 2027

RenderRank: Learning to Rerank Text with Compressed Visual Tokens

Abstract

Rendering document text as images allows vision-language models to encode documents as visual tokens, which can reduce input sequence length compared with text input. This reduction in input length is particularly useful for reranking, where each query involves scoring multiple candidate documents and token savings apply to each candidate evaluation. We introduce RenderRank, a reranker that learns query-dependent relevance scoring from compressed visual document representations instead of the text token sequences used by conventional text-based rerankers. Training first aligns relevance scores from visual inputs with those of a text-based teacher, then refines the relative scores of positive and negative documents for the same query. Across 11 datasets from BEIR, RenderRank uses 16.5–35.5% fewer input tokens while achieving an average NDCG@10 of 55.96, outperforming all evaluated text-based baselines below 4B parameters and some larger models. Across four long-document datasets, it achieves an average NDCG@10 of 88.27 with approximately half the average input token count of the evaluated text-based rerankers. In this setting, RenderRank delivers the highest average throughput of the evaluated baselines. These results demonstrate that compressed visual representations can support accurate document relevance scoring, providing an alternative to text token representations for reranking.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Reranking, Information Retrieval, RAG, Rendering, Visual Compression

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Vj8YOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The authors introduce the use of visual tokens as a representation compression mechanism used to rerank retrieval results in RAG systems in QA applications. Experiments are performed on reference text sets in QA tasks.

Critical strengths

They propose a representation compression mechanism in QA systems. Compressing relevant information in a conversation with a natural language engine is essential to keep the context as complete as possible to ensure the quality of the conversation.

Critical weaknesses

The authors sell the use of visual tokens as a "natural" mechanism for token representation compression, but there is no real estimate of the compression rate achieved with the visual representation. In the experimental evaluation, the average number of tokens used according to different experimental configurations has been reported, but it is not shown the average number of textual tokens (as a reference). It would be useful to shown the actual compression rate of visual tokens.

In this direction, instead of using only Qwen3-Reranker-4B as the reference teacher, the different textual baselines "teacher" (t(q,d)(t(q,d) should be compared with the version with visual token.

The compression due to the visual token is due to the resolution of 96 DPI. This value is not a realistic resolution in a scenario of text recognition in document images (which requires a minimum of 150DPIs to have reasonable OCR results), so it is even more relevant to assess the impact of the compression factor of the visual token in relation to textual tokens usiing realistic DPIs values.

In this sense, instead of extracting visual tokens from an image generated by an arbitrary text rendering mechanism, the authors could have learned these "visual" tokens from an implicit representation of R(d)R(d) without the need for an explicit rendering of text and aligned it with the teacher t(d,d)t(d,d) of the Eq. (1)

Questions for authors
  • In line with the elements discussed in the previous section. What is the real compression factor of visual tokens in relation to textual tokens?

  • How to apply the proposed scheme to images of scanned documents at normal resolutions (from 150 to 300 DPIs)

10 comments
AuthorsOpenReview ↗

Response to Reviewer Vj8Y (1)

We appreciate your recognition of the value of representation compression and your positive assessment of our methodology. Below, we provide additional results on token compression and teacher choice, and address concerns about rendering resolution and document representations by clarifying the distinction between rendering text and recognizing scanned documents.

 

W1/Q1

The table below reports token counts and reductions for each dataset, comparing text inputs with images rendered at the default 12pt font size using the same fully trained RenderRank model.

DatasetText tokensImage tokens (12pt)Token reduction
ArguAna660.36518.3521.5%
Climate-FEVER419.50283.3332.5%
DBPedia159.74128.8219.4%
FiQA450.74280.6237.7%
FEVER294.37197.2033.0%
HotpotQA189.73150.5220.7%
NFCorpus439.99278.6836.7%
SCIDOCS344.09257.0725.3%
SciFact454.55292.1535.7%
TREC-COVID425.95284.1333.3%
Touché-2020880.26519.9240.9%
Average429.03290.0732.4%

Image inputs reduce the number of input tokens on every dataset, with the average input length decreasing from 429.03 to 290.07—a 32.4% reduction. These counts cover the full input, including the query and instructions. For image inputs, we count both text and visual tokens.

 

W2

The training approach is effective with a different teacher family and with self-distillation. We used either Llama-Nemotron-Rerank-1B-v2 or the initial Qwen3-VL-Reranker-2B with text inputs as a frozen Stage 1 teacher, retaining the same student initialization and Stage 2 training procedure. The table below reports NDCG@10 across 11 BEIR datasets for the text-input reference models and the visual-input students after both training stages.

ModelAACFVDBPFQAFVRHQANFCSDSFTCTCHAVG
Llama-Nemotron-Rerank-1B-v2 (teacher, text input)54.2927.3745.1347.1187.6680.3438.1421.5379.8283.1432.4754.27
Qwen3-VL-Reranker-2B (initial, text input)57.3132.1045.3343.2286.8577.8637.4720.7177.3982.8735.1754.21
RenderRank (Qwen3-Reranker-4B teacher)69.1135.9146.1942.6688.9278.6037.3120.7577.0084.8234.2755.96
RenderRank (Nemotron teacher)67.3629.9844.8541.4787.5477.7636.8619.9876.9482.5031.9654.29
RenderRank (self-distillation)68.2132.3745.3241.4187.5576.5436.4620.4276.5083.1034.9454.80

The final model trained with the Nemotron teacher achieves an average NDCG@10 of 54.29, improving on the initial model's image-input score of 50.20 by 4.09 points and reaching performance comparable to the teacher's 54.27. The self-distilled model achieves 54.80, exceeding the initial model by 4.60 points with image inputs and 0.59 points with text inputs. These results show that effective transfer of relevance signals to visual inputs is not limited to the original 4B Qwen teacher: both a teacher from a different model family and the initial model's own text-based relevance signals support high reranking performance.

AuthorsOpenReview ↗

Response to Reviewer Vj8Y (2)

W3/Q2

In our setup, 96 DPI is a rendering parameter that converts font sizes from points to pixels, not a scanning resolution. At 96 DPI, a 12pt font corresponds to 16 pixels, and documents are rendered at a width of 896 pixels and a maximum height of 896 pixels. Qwen's visual preprocessing operates on pixel values and image dimensions; changing only the DPI metadata while keeping the pixel array unchanged does not alter the model input or the number of visual tokens. Rendering directly from source text allows us to control character size in pixels and information density to balance performance and token efficiency. The rendering experiments in the paper support this design choice, which is distinct from DPI requirements for scanned documents. Scanned documents at 150–300 DPI constitute a different input setting from the text rendering studied here.

 

W4

Learning compressed latent representations directly from text is a valid alternative, but it addresses a different research question. RenderRank investigates whether an existing VLM's ability to represent text visually can be used for document relevance scoring without jointly training a new compressor. We therefore freeze the visual encoder and feature mergers and train the language model using Cross-Modal Relevance Distillation and Query-Local Relevance Discrimination.

Explicit rendering serves a specific purpose: it converts text documents into inputs that the existing visual encoder can process and allows input length to be controlled through rendering density. Replacing this step with a learned latent compressor would change both the document representation and the components being trained. RenderRank does not rule out that alternative; it offers a distinct approach that reuses existing visual representations to make more document content available for relevance scoring within a limited input budget.

   

We hope that these responses and additional results clarify the scope of our work and address your concerns.

I thank the authors for their responses. However, I believe my main concern has not been addressed: the visual compression factor appears to be determined by the image resolution, so the reported compression stems largely from the initial choice of rendering at 96 DPI (16 px characters). My questions were intended to help clarify these doubts; unfortunately, the responses have raised more questions than they have answered.

  • Q1 (compression factor). The 32.4% reduction includes the query and the instructions, and therefore does not measure the compression of the document itself, as requested in my original question. This is confirmed by the variability across datasets (19–41%), which grows with document length. The only document-level figure appears in the response to another reviewer (XeLq, Q3) and corresponds to a single TREC-COVID document. This value should have been reported for all datasets and at different resolutions (DPI).

  • Q2 (resolution). The distinction between rendering DPI and scanning DPI is not a substantive one: in both cases, DPI sets the mapping between physical size and pixels, and 12 pt at 96 DPI is a low resolution compared with those typical of scanned documents (150–300 DPI). Since the number of visual tokens grows roughly with pixel area, moving from 96 to 300 DPI without downscaling would multiply this area by ~10, and the current compression could turn into an expansion.

Reviewer_AdgMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RenderRank renders text documents as images and uses the resulting visual tokens for relevance scoring with a text query. It freezes the visual encoder and feature mergers and adapts the language decoder in two stages: distillation from a text reranker, followed by ranking training on positive and negative candidates. Experiments on 11 BEIR datasets and four long-document datasets show competitive ranking quality with fewer input tokens on average and higher throughput when visual features are precomputed.

Critical strengths
  1. The paper shows that existing visual representations can support text reranking with compressed document inputs. The long-document results are encouraging: with precomputed visual features, RenderRank achieves ranking quality comparable to strong text baselines and 1.70×1.70\times the GPU forward throughput of the fastest evaluated baseline.

  2. The analyses help explain the adaptation and compression choices. The first training stage improves visual-input NDCG@10 from 50.20 to 54.86, while the font-size experiments show how compression affects accuracy and throughput.

Critical weaknesses
  1. The reported adaptation relies on a stronger external teacher in Stage 1. It remains unclear how well the method works when supervision must come from the model being adapted. For an approach that reuses an existing model, this is a relevant setting: a stronger teacher suited to the task may not always be available.

  2. The evaluation does not explicitly separate in-domain and out-of-domain results relative to the adaptation training. Table 9 includes sources such as FiQA and MLDR that also appear in evaluation. Although per-dataset results are reported, the aggregate scores do not distinguish performance on sources represented in adaptation training from performance on held-out sources.

Questions for authors
  1. Have you tried using a frozen copy of the initial Qwen3-VL-Reranker-2B in text mode as the Stage 1 teacher, while keeping the student initialization and Stage 2 unchanged? Could you compare this self-distillation setting with the current 4B-teacher setup and the initial model's text-mode performance?

  2. Could you define the ID/OOD distinction relative to both adaptation stages and identify which evaluation sources are held out? Please also report the existing ranking-quality and token-reduction results separately for the seen and held-out groups.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AdgM (1)

We appreciate your recognition of the potential of visual representations for text reranking and the encouraging long-document results. Our responses focus on the choice of teacher and generalization across training-data sources.

 

W1/Q1

Training student rerankers with relevance signals from strong teachers is an established approach in recent Qwen-based research. A study using a Qwen2.5-based 7B teacher to train smaller rerankers [1] analyzes the benefits of distillation from a strong teacher, while Yuanbao-Reranker [2] distills preferences between candidates from the relevance scores of Qwen3-Reranker-8B. We likewise use Qwen3-Reranker-4B to transfer relevance signals obtained from text inputs to compressed visual inputs. Using a strong teacher to provide high-quality relevance supervision is therefore consistent with established distillation practice.

To test this, we conducted a self-distillation experiment using the initial Qwen3-VL-Reranker-2B with text inputs as a frozen Stage 1 teacher. We retained the same student initialization and Stage 2 training procedure. The table below compares NDCG@10 for the initial model with text inputs and the fully trained models with image inputs.

ModelAACFVDBPFQAFVRHQANFCSDSFTCTCHAVG
Qwen3-VL-Reranker-2B (initial, text input)57.3132.1045.3343.2286.8577.8637.4720.7177.3982.8735.1754.21
RenderRank (self-distillation, image input)68.2132.3745.3241.4187.5576.5436.4620.4276.5083.1034.9454.80
RenderRank (4B teacher, image input)69.1135.9146.1942.6688.9278.6037.3120.7577.0084.8234.2755.96

The final model trained with self-distillation achieves an average NDCG@10 of 54.80, exceeding the initial model by 4.60 points with image inputs (50.20) and 0.59 points with text inputs (54.21). These results show that our two-stage training can improve visual-input reranking using the initial model's own text-based relevance signals, without a stronger external teacher. Using the original 4B teacher increases performance to 55.96: a strong teacher provides additional gains, rather than being a prerequisite for the approach.

[1] Distillation versus Contrastive Learning: How to Train Your Rerankers.
[2] Yuanbao-Reranker: Document-Grounded Synthesis and Preference Distillation for Text Reranking.

AuthorsOpenReview ↗

Response to Reviewer AdgM (2)

W2/Q2

Supervised training on queries and relevance annotations from public retrieval datasets is used for both embedding models and rerankers. Multilingual E5 uses datasets including FEVER and HotpotQA for fine-tuning [1], while BGE-M3 trains on the MLDR training split and evaluates on its test split [2]. For rerankers, the Qwen3 Embedding/Reranker technical report lists MS MARCO, NQ, HotpotQA, and MLDR as supervised training sources and evaluates rerankers on retrieval tasks including MLDR [3]. The RLHN study also trains a Qwen2.5-3B-based reranker on BGE training data containing FiQA, FEVER, and HotpotQA, and evaluates it on BEIR [4]. Sharing a dataset source is distinct from training on evaluation examples. We define Seen (ID) sources as those used in at least one of the two adaptation stages, and Held-out (OOD) sources as those used in neither stage. The table below reports source inclusion at each stage, alongside the final model's ranking performance and token reduction.

DatasetStage 1Stage 2RenderRank NDCG@10Text input tokensImage input tokensToken reduction
Seen (ID)
ArguAnaXO69.11660.36518.3521.50%
FiQAOO42.66450.74280.6237.74%
FEVEROO88.92294.37197.2033.01%
HotpotQAOO78.60189.73150.5220.67%
SCIDOCSXO20.75344.09257.0725.29%
Held-out (OOD)
Climate-FEVERXX35.91419.50283.3332.46%
DBPediaXX46.19159.74128.8219.36%
NFCorpusXX37.31439.99278.6836.66%
SciFactXX77.00454.55292.1535.73%
TREC-COVIDXX84.82425.95284.1333.29%
Touché-2020XX34.27880.26519.9240.94%

Input tokens are reduced on all six held-out datasets, with reductions ranging from 19.36% to 40.94%. The token-efficiency benefits of visual representations therefore extend beyond the training sources to diverse retrieval tasks.

[1] Multilingual E5 Text Embeddings: A Technical Report.
[2] M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.
[3] Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models.
[4] Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs.

 

We hope that our responses and additional experimental results address your concerns.

AuthorsOpenReview ↗

Dear Reviewer AdgM

Our response includes additional self-distillation experiments and dataset-level results for seen and held-out sources to address your concerns about the teacher’s role and generalization. We kindly ask you to review these additions and reconsider your assessment in light of the additional evidence.

Please let us know if any concerns remain so that we can address them during the discussion. Thank you for your time and consideration.

Reviewer_AdgMOpenReview ↗

Response

Thank you for the detailed response and the additional experiments.

The self-distillation experiment answers my first question. With the initial Qwen3-VL-Reranker-2B in text mode as the Stage 1 teacher, the image-input model improves from 50.20 to 54.80, slightly above the initial model's text-mode score (54.21). The approach therefore does not depend on a stronger external teacher, and the 4B teacher adds a further gain. The Nemotron-teacher result supports the same conclusion.

The seen/held-out breakdown answers my second question, and token reduction holds on all held-out sources. One observation from your tables: relative to the initial model's text-mode scores, the gains of the final model are concentrated on seen sources (about +2.8 NDCG@10 on average, largely from ArguAna), while held-out sources are roughly on par (about +0.9). This is consistent with the paper's main claim of approaching text-mode quality with fewer tokens, but I would suggest reporting the group averages in the paper and stating this explicitly.

My main concerns have been addressed, and I will raise my rating to 3.

Reviewer_XeLqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about reranking in the context of RAG. To address the limitations of long text token sequences, the authors propose rendering document images as visual tokens to reduce input length. Training is carried out in two stages: cross-modal relevance distillation and query-local relevance discrimination using InfoNCE. An experimental validation is done on 11 datasets from BEIR and four-long document datasets. Ablations cover both training stages, font size and visual-embedding caching.

Critical strengths
  • The idea of converting each candidate document into an image for re-ranking is interesting.
  • The paper is well written, and the motivation is clear.
  • The distillation at the score level is a good point
  • The experimental validation is well designed, including ablation studies.
  • The computational efficiency using Peng approximation is a good point.
  • The paper provides useful ablations.
Critical weaknesses
  • The technical novelty is limited: each component is an established technique, and the contribution is mainly their combination and an empirical characterization
  • The paper is not evaluated on native document images (Visual RAG), yet this is where the approach is most attractive.
  • Variance is not reported.
Questions for authors
  • How does the proposed approach generalize to other languages?
  • How does the proposed approach compare with the approaches mentioned in Appendix 1 ?
  • What is the storage cost of the visual embedding ?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XeLq (1)

We appreciate your recognition of visual representations for reranking and score-level distillation, along with your positive assessment of the motivation, ablations, and computational efficiency analysis. Below, we clarify our contribution and its distinctions from related work, and address your concerns and questions with additional experimental evidence.

 

W1

Our central contribution is to extend an existing VLM's document understanding capabilities to query-dependent relevance scoring, using compressed visual representations to rerank text documents. Reading document content visually and assigning query-dependent relevance scores that distinguish candidate priorities are different learning objectives. RenderRank addresses this gap through Cross-Modal Relevance Distillation and Query-Local Relevance Discrimination while keeping the visual encoder frozen. Average NDCG@10 with image inputs improves from 50.20 for the initial model to 54.86 after Stage 1 and 55.96 after Stage 2, demonstrating the contribution of relevance training beyond simply changing the input format.

To assess the benefits under constrained input budgets, we added comparisons with matched maximum input lengths during revision. The tables below report NDCG@10 on 2WikiMQA, QMSum, and SummScreenFD with each model's maximum input length set to 2K, 4K, or 8K.

 

2WikiMQA

Model2K4K8K
Qwen3-Reranker-0.6B85.5591.3592.80
Qwen3-Reranker-4B86.9591.1893.10
zerank-284.0590.8192.91
LightOn-rerank-PW-4B84.1588.9291.55
RenderRank91.1693.4294.30

 
 
QMSum

Model2K4K8K
Qwen3-Reranker-0.6B43.9847.2751.77
Qwen3-Reranker-4B45.1949.0554.62
zerank-245.1649.2954.86
LightOn-rerank-PW-4B45.4748.7454.61
RenderRank48.8954.6559.15

 
 
SummScreenFD

Model2K4K8K
Qwen3-Reranker-0.6B95.4298.0298.17
Qwen3-Reranker-4B97.5498.6399.15
zerank-298.1698.8398.85
LightOn-rerank-PW-4B98.1299.0398.94
RenderRank98.8099.0199.00

RenderRank achieves the highest performance among the compared models at all three input limits—2K, 4K, and 8K—on both 2WikiMQA and QMSum. On 2WikiMQA, its 4K score of 93.42 exceeds the best baseline score at 8K (93.10). On QMSum, RenderRank reaches 54.65 at 4K, matching the performance of Qwen3-Reranker-4B at 8K (54.62) with half the maximum input budget. On SummScreenFD, it achieves the highest score at 2K (98.80) and maintains scores around 99 at 4K and 8K. These results demonstrate the benefits of compressed visual representations for effective long-document reranking under smaller input budgets.

 

W2

Our work concerns reranking text documents represented as visual tokens, rather than Visual RAG over native document images. Rendering reduces input sequence length so that more document content can be used within a limited context window. The contribution is that the token efficiency of visual representations can also be applied to ordinary text documents, not just documents already available as images. Comparisons on text retrieval benchmarks therefore align with our research objective; evaluation on native document images is a separate extension.

 

W3

If the concern refers to evaluation variability, we use a fixed model and fixed query–candidate sets to determine relevance scores and rankings. This procedure involves neither stochastic generation nor candidate sampling, so we do not expect variation in reranking performance from repeating evaluation with the same model, inputs, and evaluation settings.

AuthorsOpenReview ↗

Response to Reviewer XeLq (2)

Q1

Many non-English languages generally require more text tokens than English to express equivalent content. In contrast, the number of visual tokens depends on the rendered image dimensions and the visual encoder's patch and merging configuration, rather than the text tokenizer's segmentation. If documents can be represented within the same visual-token budget, these languages can therefore achieve greater token reduction than English, making this approach useful for incorporating more document content within a limited input length. Training and evaluation in this study were conducted on English documents.

 

Q2

Related approaches differ in where compression occurs and which components are trained.

RRK [1] jointly trains a separate text compressor and a reranker, representing each document with eight memory tokens as input to the reranker. It thus reduces the reranker's input itself and also evaluates long documents with input lengths up to 2,048 tokens. CompRank [2] independently computes document representations, selects document KV states attended to by the query in the final block, and aggregates attention weights to score relevance. Rather than replacing document encoding with a shorter input, it reduces the tokens involved in query–document interactions.

LTC [3] processes the original text token sequence in the early layers, then applies pooling at an intermediate layer to shorten the sequence processed by subsequent layers. Its experiments include long-document generalization and applications to generative listwise rerankers. Tevatron-Elastic [4] is a training framework that combines such intermediate-layer token compression with adjustable layer counts and output embedding dimensions.

In contrast, RenderRank represents rendered documents using a frozen VLM visual encoder and supplies a short visual-token sequence from the first layer of the language model. It learns query-dependent relevance scoring from existing visual representations without jointly training a separate text compressor or selecting and merging tokens within the language model. The distinction is therefore not simply whether a method handles long documents, but how it reduces the input and what it trains.

[1] Efficient Listwise Reranking with Compressed Document Representations
[2] CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring
[3] Layer-wise Token Compression for Efficient Document Reranking
[4] Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers

 

Q3

Here, embeddings refer to the final image embeddings extracted from rendered document images. We used one fixed document from TREC-COVID and rendered increasingly long prefixes of 128, 256, 384, … text tokens. Each visual token has a 2,048-dimensional final embedding, requiring 4 KiB when stored in BF16.

Text tokensVisual tokensFinal visual embedding storage (KiB)
12884336
256140560
384196784
512224896
6402801,120
7683361,344
8963921,568
1,0244761,904
1,1525322,128
1,2805882,352
1,4086722,688
1,5367282,912
1,6647843,136
1,7928683,472
1,9209243,696
2,0489803,920
2,1761,0364,144
2,3041,1204,480
2,4321,1764,704
2,5601,2605,040

Just as retrieval systems build document collections and indexes in advance, document-level visual representations can be computed ahead of time, stored by document ID, and loaded for the retrieved candidates. With a fixed model and rendering configuration, a document's representations can be reused across queries without recomputation. The table reports the storage required for the final visual embedding tensors, which scales linearly with the number of visual tokens in BF16.

 

If these clarifications and additional results address your concerns, we would appreciate a reassessment of our contribution in light of this evidence.

AuthorsOpenReview ↗

Dear Reviewer XeLq

Building on the aspects you positively assessed, including the motivation, score-level distillation, and ablation studies, our response further clarifies the contribution relative to prior work and presents additional long-document results under matched input budgets. We would be grateful if you could revisit our response and reconsider your assessment in light of these clarifications and results.

Please let us know if any concerns remain so that we can address them during the discussion. Thank you for your time and consideration.

Reviewer_oT8POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents RenderRank, a document reranking framework that renders text documents as compressed visual tokens to improve efficiency. Documents are rendered into images, encoded by a frozen vision encoder, and scored jointly with textual queries using a language decoder initialized from Qwen3-VL-Reranker-2B. The training protocal includes two stage: (1) Knowledge distillation to enforce a image-based student to align the ability of text-based teachers; (2) InfoNCE contrastive learning to refine the prediction by restric the order of candidates within each query. Experiments on 11 BEIR datasets show competitive reranking performance with 16.5-35.5% fewer input tokens than the evaluated text-based baselines. On four long-document datasets, RenderRank achieves comparable or slightly better ranking performance while approximately halving input token counts and achieving 1.70x the throughput of the fastest evaluated baseline under precomputed visual-feature settings.

Critical strengths
  1. Clear and well-motivated problem. Reducing document token counts is important for a wide range of applications, particularly those involving long documents. Rendering text as images offers a potential approach to achieving more compact document representations.
  2. Detailed analysis of rendering configurations. The paper systematically investigates the effects of rendering configurations, providing empirical evidence to support its design choices.
  3. Simple and effective methodology. The proposed two-stage training strategy is straightforward and easy to understand, with its effectiveness supported by the ablation studies in Tables 6 and 15.
Critical weaknesses
  1. The actual benefit of visual compression over text-based alternatives is not convincingly demonstrated. The main experiments lack a direct accuracy–efficiency comparison between the text-input and image-input variants of the same model. Table 15 further shows that the text-input variant achieves higher ranking accuracy, raising questions about whether visual compression provides a sufficiently favorable trade-off compared to the text-input counterpart.
  2. The reported efficiency gains depend heavily on precomputed visual features. The main efficiency comparisons exclude visual encoding and preprocessing costs, which are additional overheads introduced by the image-input approach. Although caching is reasonable, the paper does not sufficiently establish whether the reported speedup holds when these costs are considered in practical settings.
Questions for authors

1. Despite the promising results, the paper does not demonstrate whether visual compression provides a favorable accuracy-efficiency trade-off over text-based alternatives. Table 15 reveals an important observation: although both training stages use image inputs, the model's performance with text inputs also improves substantially and remains higher than that with image inputs. This suggests that the training procedure itself may account for much of the improvement in ranking performance, while visual compression trades some accuracy for efficiency. If this is the case, a direct comparison between the text-input and image-input variants of the same model is essential to assess the actual benefit of visual compression. However, the main experiments compare RenderRank primarily against different text-based rerankers, without evaluating both the ranking performance and inference efficiency of its own text-input counterpart. Consequently, it remains unclear whether the proposed visual representation provides a meaningful accuracy-efficiency advantage.

2. The reported efficiency gains rely heavily on precomputed visual features, raising concerns about their practical benefits. Section 5.2.2 measures GPU throughput without accounting for preprocessing, while Table 3 also excludes visual encoding from the FLOPs calculation. Importantly, image rendering and visual encoding introduce additional costs that are not required by text-based rerankers. Although caching visual features is reasonable, its impact on efficiency is substantial. As shown in Table 6(b), throughput drops from 76.83 to 37.31 PPS without caching. However, the 1.70× speedup reported on long-document datasets (Table 4) is evaluated only with caching, and the storage and data-transfer costs of cached features are not reported. The authors should further discuss or evaluate how these additional costs affect efficiency in practical settings, and whether they could offset the speedup gained from visual compression.

3. The long-document experiments provide limited evidence of the effectiveness of visual compression under different token budgets. The performance on MLDR is nearly saturated, with all models achieving NDCG@10 scores above 98.9 (Table 4). However, MLDR is the only dataset used for the fixed-token-budget comparison in Table 5. This makes it difficult to assess whether the observed advantages of visual compression generalize to more challenging long-document tasks. It would be helpful to extend the fixed-token-budget evaluation to other long-document datasets, where the performance differences between models are more pronounced.

4. The technical novelty and advantages over closely related methods are not sufficiently established. The main components of RenderRank are largely based on existing techniques. Visual text compression has been explored in Glyph, while ZipRerank and RRK have already studied efficient reranking using rendered text or compressed document representations. The authors should better distinguish their contributions from these closely related works and, where applicable, provide comparisons with existing compression-based reranking methods.

5. The FLOPs estimation in Equation (3) appears to incorrectly account for grouped-query attention (GQA). The quadratic attention term in Equation (3) is divided by the query-to-KV head ratio (r). However, standard GQA reduces the number of KV heads without proportionally reducing the attention computation across query heads. Therefore, the quadratic term should not be divided by (r) under the usual FLOPs counting convention. This may lead to inaccurate computational estimates in Table 3, particularly when comparing models with different GQA ratios. The authors should clarify the derivation of Equation (3) and verify the reported FLOPs and QPP values.

4 comments
AuthorsOpenReview ↗

Response to Reviewer oT8P (1)

We appreciate your recognition of the importance of reducing document token counts, particularly for long documents, and your positive assessment of our systematic rendering analysis and simple, effective two-stage training. Below, we present additional experimental results and address your concerns about performance–efficiency trade-offs, computational costs, and distinctions from related work.

 

W1/Q1

Text- and image-input performance for the same fully trained model is reported in Table 15 of the submitted paper. Below, we present those scores alongside average input token counts and throughput to make the performance–efficiency trade-off explicit.

RenderRankAverage NDCG@10Average input tokensPPS
Text57.21429.0352.80
Image (12pt)55.96290.0776.83

Image inputs reduce the average input token count by 32.4% and provide approximately 1.46 times the GPU forward throughput under the measurement setting that uses precomputed visual features. The performance difference is 1.25 NDCG@10 points, illustrating how visual document compression offers a performance–efficiency trade-off within the same model. Our objective is not to claim that image inputs always outperform text inputs, but to provide high relevance-scoring performance while reducing the input budget required to represent documents. The additional evaluations in Q3 below demonstrate how these compression benefits support effective long-document reranking with smaller maximum input budgets.

 

W2/Q2

Throughput including visual encoding is already reported in Table 6(b) of the submitted paper: 76.83 PPS with cached visual representations and 37.31 PPS when visual encoding is included. The paper therefore compares both precomputed representations and visual encoding performed during inference, quantifying the effect of this cost on throughput.

Precomputing document representations moves query-independent document processing offline while retaining query-dependent relevance scoring online. As with indexing a document collection, a fixed model and rendering configuration allow rendering and visual encoding to be performed when documents are added or updated, with the resulting representations reused across subsequent queries. EDJE [1] likewise precomputes and stores visual tokens for online reranking. This avoids repeating rendering and visual encoding for every candidate evaluation and can improve online throughput in retrieval settings where documents are reused.

[1] Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking

 

Q3

We extended the fixed-input-length evaluation from MLDR to 2WikiMQA, QMSum, and SummScreenFD. The tables below report NDCG@10 with the maximum input length set to 2K, 4K, or 8K for both RenderRank and the comparison models.

 
2WikiMQA

Model2K4K8K
Qwen3-Reranker-0.6B85.5591.3592.80
Qwen3-Reranker-4B86.9591.1893.10
zerank-284.0590.8192.91
LightOn-rerank-PW-4B84.1588.9291.55
RenderRank91.1693.4294.30

 
QMSum

Model2K4K8K
Qwen3-Reranker-0.6B43.9847.2751.77
Qwen3-Reranker-4B45.1949.0554.62
zerank-245.1649.2954.86
LightOn-rerank-PW-4B45.4748.7454.61
RenderRank48.8954.6559.15

 
SummScreenFD

Model2K4K8K
Qwen3-Reranker-0.6B95.4298.0298.17
Qwen3-Reranker-4B97.5498.6399.15
zerank-298.1698.8398.85
LightOn-rerank-PW-4B98.1299.0398.94
RenderRank98.8099.0199.00

RenderRank achieves the highest performance among the compared models at all three input limits—2K, 4K, and 8K—on both 2WikiMQA and QMSum. On 2WikiMQA, its 4K score of 93.42 exceeds the best baseline score at 8K (93.10). On QMSum, RenderRank reaches 54.65 at 4K, matching the performance of Qwen3-Reranker-4B at 8K (54.62) with half the maximum input budget. On SummScreenFD, it achieves the highest score at 2K (98.80) and maintains scores around 99 at 4K and 8K. These results demonstrate the benefits of compressed visual representations for effective long-document reranking under smaller input budgets.

AuthorsOpenReview ↗

Response to Reviewer oT8P (2)

Q4

Glyph [1] uses rendered text for long-context processing and response generation, while ZipRerank [2] reranks document images through query-dependent visual token selection and listwise scoring. RRK [3] jointly trains a separate text compressor and a reranker to represent documents with a fixed number of memory tokens and rerank candidate sets using those representations. Our focus is not on learning a new compressor, but on learning query-dependent document relevance from an existing VLM's compressed visual representations and using their token efficiency for text reranking. We freeze the visual encoder and feature mergers and train the language model on visual tokens whose number varies with document length and rendering density. Unlike listwise configurations in which multiple candidates share one input budget, we process the ordered images of a long document as a single candidate, using a document-specific input budget. The additional QMSum result in Q3 shows that RenderRank matches the 8K performance of Qwen3-Reranker-4B using a 4K input budget, demonstrating effective long-document relevance scoring with a smaller input budget through compressed visual representations.

[1] Glyph: Scaling Context Windows via Visual-Text Compression
[2] Very Efficient Listwise Multimodal Reranking for Long Documents
[3] Efficient Listwise Reranking with Compressed Document Representations

 

Q5

Equation (3) follows the computational cost approximation in Section 3.2.1, Equations (9)–(10), of E²R-FLOPs [4]. Its official implementation applies the GQA query-to-KV head ratio in the same way. This coefficient therefore comes from the adopted estimation method rather than being introduced in our work. We report FLOPs and QPP as estimates based on that method, while evaluating actual inference efficiency separately through measured PPS.

[4] Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers

 
 

The additional experiments strengthen the evidence for the benefits of visual document representations under constrained input budgets. We would appreciate your reconsideration of the remaining concerns and our contribution in light of these clarifications and results.

Thank you for the additional experiments and clarifications. The same-model comparison makes the accuracy-efficiency trade-off clearer, and the results across more long-document datasets address my concern about the fixed-token-budget evaluation. Overall, I am satisfied with the response. One minor suggestion is to state explicitly alongside the 1.70× throughput result that it assumes precomputed visual features and excludes cache I/O. The rating has been updated.

AuthorsOpenReview ↗

Thank You to Reviewer oT8P for the Thoughtful Discussion

We sincerely appreciate the care with which you considered our work and additional experiments. These days, we do not take such thoughtful engagement for granted, and this exchange with you has been particularly meaningful to us. Your feedback and this discussion have given us a clearer understanding of how to improve the quality and clarity of our paper.

As suggested, we will explicitly state alongside the 1.70× throughput result that it assumes precomputed visual features and excludes cache I/O.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.