acceptodds
Under review as a conference paper at ICLR 2027

PixelRAG: Retrieval and Reading in Pixel Space over Millions of Web Screenshots

11 views

Abstract

Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting. We introduce PixelRAG, a new retrieval-augmented method that represents websites in their native visual form and performs retrieval and reading entirely in pixel space, enabling an end-to-end architecture that eliminates text abstraction. PixelRAG is, to our knowledge, the first pipeline to operate over a full Wikipedia corpus in this form, scaling to a datastore of 30 million screenshot images with an efficient visual retrieval index. Built on an existing visual embedding model (i.e., Qwen3-VL-Embedding), PixelRAG further fine-tunes this model on screenshot data with carefully curated contrastive training data. Retrieved screenshots are then fed directly as pixel inputs to a VLM, without intermediate text conversion. PixelRAG consistently outperforms both no-retrieval and text-based RAG baselines, improving accuracy by up to 18.1% over text-based baselines, most surprisingly on widely studied text-centric tasks such as NQ and SimpleQA. It also achieves strong gains on multimodal open-domain QA (e.g., MMSearch), benchmarks over noisy news corpora (e.g., LiveVQA), and agentic benchmarks (e.g., MoNaCo). Finally, pixel representations enable an efficiency lever for RAG through image compression, effectively reducing reader input tokens by up to 3× while maintaining accuracy. Our results question the necessity of text representations in web retrieval, suggesting that web RAG can operate in the web's native visual form while improving accuracy and reducing reader input tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual RAG, Multimodal Retrieval, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zEu7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for web RAG that avoids parsing HTML into text. Instead, it renders web pages into screenshots and divides them into fixed-size image tiles. Each tile is encoded using Qwen3-VL-Embedding, and the retrieved image tiles are then fed into a vision-language model for answer generation. The authors construct a screenshot-based index from web-scale Wikipedia data. They also synthesize queries based on webpage screenshots and fine-tune the embedding model through contrastive learning. On existing QA benchmarks (e.g., NQ and SimpleQA), this method outperforms text-based RAG approaches that rely on HTML parsers.

Critical strengths

One strength is that the paper proposes replacing the approach of cleaning webpage HTML into plain text with rendered webpage screenshots as the retrieval representation for large-scale web RAG. On existing QA benchmarks (e.g., NQ and SimpleQA), this method outperforms text-based RAG approaches using HTML parsers. This work challenges the default use of text representations in web RAG. Another strength is that the retrieval corpus used in the experiments is large-scale, consisting of a Wikipedia snapshot. The experimental results on this web-scale index demonstrate that the method's effectiveness is not limited to small document collections.

Critical weaknesses

One major weakness is that the method is similar to the pipeline of existing works such as VisRAG “VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents” Shi Yu. et.al. 2024. involving page image encoding, retrieval, and answer generation using a vision-language model. This work simply extends the existing paradigm of multimodal document RAG (e.g., VisRAG) to large-scale web data.

Questions for authors

What are some critical advantages over the existing work VisRAG?

1 comment

We thank the reviewer for recognizing that the work challenges the default text representation in web RAG and that the evaluation runs at web scale.


W1 / Q1. Critical advantages over VisRAG

Reviewer: "The method is similar to the pipeline of existing works such as VisRAG ... This work simply extends the existing paradigm of multimodal document RAG (e.g., VisRAG) to large-scale web data. (Q1) What are some critical advantages over the existing work VisRAG?"

The key difference from VisRAG is the setting: VisRAG focuses on visually rich document retrieval, while PixelRAG studies web retrieval for predominantly text-centric tasks. While the benefits of visual retrieval on visually rich documents are largely expected, our surprising finding is that visual retrieval also significantly outperforms text retrieval on traditional text-centric tasks, where text has long been the default representation. This shift introduces three important differences:

  1. Benchmarks. VisRAG evaluates on visually rich document QA benchmarks such as ChartQA, PlotQA, and SlideVQA, where visual information is central to answering questions. In contrast, PixelRAG evaluates on traditional text-centric benchmarks such as NQ and SimpleQA, where text retrieval has long been the standard. PixelRAG improves NQ R@3 from 45.8 to 58.8 and SimpleQA accuracy from 71.6 to 78.8 over text RAG.
  2. Scale. VisRAG focuses on relatively small document collections(thousands of pages), while PixelRAG scales to a complete open-domain knowledge base of 7.1M Wikipedia articles and 30M tiles. To our knowledge, this is the first study of visual retrieval at the scale of a full open-domain knowledge base.
  3. Method and Insights. Beyond demonstrating this surprising result, we systematically investigate why pixels outperform text, identifying structural information lost during HTML parsing and keyword-dense text chunks that crowd out relevant evidence (Tables 2–3, Appendix D).

In summary, VisRAG demonstrates that pixels help when visual information is essential; PixelRAG shows that pixels help even on traditional text-centric tasks and at the scale of a complete open-domain knowledge base. We will clarify these distinctions in the revised paper.

Reviewer_MrguOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PixelRAG presents a retrieval-augmented generation framework that retrieves webpage screenshots instead of parsed text. It scales to approximately 30M Wikipedia screenshots tiles and fine-tunes a visual embedding model using synthetic queries and hard-negative mining. The authors demonstrate improvements over text-based RAG across several QA benchmarks.

Critical strengths

The paper addresses an important limitation of text-based RAG where HTML parsing can discard layout and structural information. Scaling screenshot-based retrieval to millions of web pages is a meaningful contribution. The evaluation setup is promising, covering multiple QA benchmarks, retrieval configurations, and reader models. The ablations on retrieval modality, contrastive training, and image compression are useful and provide reasonable evidence supporting the approach.

Critical weaknesses

My main concern is the limited architectural novelty. Previous works, particularly VisRAG (ICLR 2025), VDocRAG (CVPR 2025), and Document Screenshot Embedding (EMNLP 2024), have already explored related visual retrieval architectures. Although some are cited, the distinctions are not sufficiently discussed, and VDocRAG appears to be missing. The primary contribution seems to be scaling an existing paradigm rather than proposing a fundamental new approach.

Second, the improvements over stronger text baselines are less substantial than the headline results suggest. Ex: improvement over Trafilature are only 2.8 points on NQ and 3.6 points on MMSearch. The paper present little comparisons with structure preserving alternatives such as HtmlRAG, as evaluating raw HTML alone is not sufficient.

Finally, the efficiency claims are somewhat overstated. Although PixelRAG reduces reader input tokens, it requires significantly more storage and has higher end to end latency than text based RAG. The generalisation of the fine-tuned reliever to other webpage distributions also remain uncertain.

Tanaka et al., “VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents,” CVPR 2025, pp. 24827–24837.

Ma et al., “Unifying Multimodal Retrieval via Document Screenshot Embedding,” EMNLP 2024, pp. 6492–6505.

Yu et al., “VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents,” ICLR 2025, pp. 21074–21098.

Questions for authors
  1. What fundamentally distinguishes PixelRAG from VisRAG and VDocRAG beyond scaling, and why is VDocRAG not discussed?
  2. Could the authors compare against structure-preserving approaches such as HtmlRAG and stronger text retrieval pipelines?
  3. Are the smaller improvements statistically significant across all benchmarks?
  4. How does PixelRAG compare with text-based RAG when considering total inference latency, memory, and preprocessing costs?
3 comments

We thank the reviewer for the detailed review and for recognizing the problem, the scale, and the ablations.


W1 / Q1. Difference from VisRAG, VDocRAG, and DSE

Reviewer: "What fundamentally distinguishes PixelRAG from VisRAG and VDocRAG beyond scaling, and why is VDocRAG not discussed?"

The key difference is that we study web retrieval rather than visually rich document retrieval, leading to qualitative differences in scale, evaluation benchmarks, and why visual retrieval helps. While the benefits of visual retrieval on visually rich documents are relatively intuitive, our surprising finding is that visual retrieval significantly outperforms text retrieval even on traditionally text-centric tasks, where text has long been the default representation.

Benchmarks VisRAG evaluates on visually rich document QA benchmarks such as ChartQA, PlotQA, and SlideVQA, where visual information is central to answering questions. VDocRAG similarly focuses on visually rich documents. In contrast, we show that visual retrieval outperforms strong text baselines even on well-established text-centric benchmarks such as NQ and SimpleQA. The closest prior work, DSE (Unifying Multimodal Retrieval via Document Screenshot Embedding), uses only a subset of Wikipedia pages and the top portion of each page and focuses on retrieval rather than end-to-end RAG.

Insight. We systematically show that text parsing loses important structural information, including tables, infoboxes, and page layout, while keyword-dense text chunks can crowd out relevant evidence.

Scale. We evaluate on all 7.1M Wikipedia articles (30M tiles), far beyond the thousands of documents used in prior visual RAG work. This establishes visual retrieval’s effectiveness on a comprehensive open-domain knowledge base, and we will release the full index and pipeline to enable future research at this scale.

In short, PixelRAG does not simply scale up existing visual document RAG. It establishes a surprising new empirical finding in a qualitatively different retrieval setting, explains why visual representations outperform text even on text-centric tasks, and makes visual retrieval practical at web scale. We thank the reviewer for pointing out the missing VDocRAG reference and will add it to the related work.


W2 / Q2. HtmlRAG and stronger text baselines

Reviewer: "Could the authors compare against structure-preserving approaches such as HtmlRAG and stronger text retrieval pipelines?"

We thank the reviewer for raising this point. Our main baseline, Trafilatura, already preserves tables and infoboxes as Markdown and performs best among seven parsers (Appendix D.1). However, the reviewer raises a valid concern that stronger parsing methods might better preserve page structure. To address this, we evaluate two neural extractors, Dripper and ReaderLM-v2:

ExtractorR@1 SQA / NQ / NQTExtraction time (7.1M articles)
Trafilatura (paper baseline)67.7 / 31.5 / 25.93.5 h, CPU
Dripper68.7 / 32.1 / 29.52,370 h, H200
ReaderLM-v264.5 / 26.9 / 23.34,110 h, H200
PixelRAG76.3 / 45.5 / 42.3no extraction

Keeping raw HTML makes things worse. The paper tests two HTML pipelines. Giving the reader the original HTML drops SimpleQA accuracy from 71.6 to 59.8 (Appendix D.5). A fully HTML-native pipeline, which indexes and reads raw HTML, loses 29.4 points on NQ and 24.1 on NQ-Tables (Appendix D.6). Markup inflates the context 3.8×.

On HtmlRAG. HtmlRAG is similar in spirit to our Text → HTML baseline (Table 3, Appendix D.5). In both, the evidence is selected with text, and HTML is used only as the reader's input. HtmlRAG gets pages from Bing search, then prunes their HTML by ranking blocks on the text-embedding similarity of their plain text. Its pruning addresses the token cost we measure in D.5, but the selection is still text-based. Most of our gain comes from selecting evidence in pixel space: with the reader input held to text, screenshot retrieval still beats text retrieval (72.2 vs. 71.6 on SimpleQA, 64.5 vs. 59.0 on LiveVQA; Table 3). We compared against HtmlRAG (Tan et al., WWW 2025), using its released cleaning and pruning models and the same reader. HtmlRAG improves over raw HTML on SimpleQA (65.0 vs. 59.8 in Appendix D.5) but is at best on par with plain text (SimpleQA 65.0 vs. 71.6, NQ 56.0 vs. 55.9, NQ-Tables 40.6 vs. 42.5 in Table 1), and it stays below PixelRAG on all three (78.8, 58.7 and 48.8 in Table 1).

Stronger text retrieval. Furthermore, motivated by the reviewer’s suggestion, we additionally evaluate a stronger hybrid retrieval baseline combining BM25 and dense retrieval, which achieves R@1 of 73.2 / 41.1 / 38.3 (SQA / NQ / NQT). We also compare against a text retriever trained with our recipe (73.3 / 39.0 / 33.6). PixelRAG consistently outperforms both, achieving 76.3 / 46.4 / 42.4.


Q3. Statistical significance

Reviewer: "Are the smaller improvements statistically significant across all benchmarks?"

The gains over Trafilatura are significant on NQ, NQ-Tables, SimpleQA, EVQA and LiveVQA under an exact McNemar test on per-question correctness, with both systems answering the same questions:

BenchmarkQuestionsAcc gainMcNemar p
NQ1,000+2.80.017
NQ-Tables1,067+6.35 × 10⁻⁷
SimpleQA946+7.23 × 10⁻⁵
EVQA568+15.51 × 10⁻¹⁴
LiveVQA26,888+11.0< 10⁻²⁰⁰

Acc gain is from Table 1; the p-values use the same configuration as Table 1.

On the smaller gains:

  • NQ. On a benchmark this saturated, the 2.8-point gain is significant (p = 0.017), and on retrieval it is large (R@3 45.8 → 58.8), which matters on its own because users see the retrieved evidence. NQ answers are mostly plain prose, the best case for text extraction.
  • MMSearch. It covers general web topics, but we retrieve from Wikipedia only, so many answers are not in the corpus at all.
  • Consistency. In Table 1 and Figure 2, the gain holds for every benchmark, four readers, and every k. In our 29-model sweep it holds for 15 of the 16 newer readers, with one tie (Table 9).

W3 / Q4. Latency, memory, and preprocessing cost

Reviewer: "How does PixelRAG compare with text-based RAG when considering total inference latency, memory, and preprocessing costs?"

Memory/Storage. PixelRAG does not require storing the rendered screenshots. With on-demand rendering, it only needs an additional 120 GB retrieval index, compared to a 64 GB index plus 52 GB of source text for text RAG.

Latency. We thank the reviewer for pointing out that our original analysis reported input token counts rather than actual inference latency. We therefore conducted additional measurements comparing PixelRAG (2 image tiles) against text RAG (top-3 text chunks). Retrieval: PixelRAG searches over more vectors, resulting in slightly higher GPU retrieval latency (14.0 vs. 10.5 ms). Reader: At matched accuracy, PixelRAG requires only 1,707 image tokens versus 2,664 text tokens (36% fewer). This translates into lower prefill latency (274 vs. 286 ms) and 33% higher throughput (5.6 vs. 4.2 requests/s at concurrency 32). Overall: Combining retrieval and reader prefill, PixelRAG achieves 288 vs. 297 ms, demonstrating that visual retrieval can be more efficient despite its slightly higher retrieval overhead. We will include these new measurements in the revised paper.

Preprocessing. Rendering and embedding all 7.1M Wikipedia articles takes approximately two days on one machine. Text extraction ranges from 3.5 hours to 4,110 GPU-hours depending on the parser, plus embedding.

The rebuttal addresses most of my concerns. Therefore, I maintain my original recommendation. Weak Accept.

Reviewer_JMUZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper describes a visual document retrieval pipeline for Retrieval-Augmented Generation (RAG) in large-scale scenarios such as Wikipedia. The proposed framework follows a classical RAG architecture, leveraging standard methods for indexing, embedding, and retrieval. The paper introduces a procedure to generate training data for contrastive learning of the visual document embeddings. Through extensive experimentation and ablation studies, the paper shows that the proposed visual retrieval framework outperforms standard text-based retrieval baselines.

Critical strengths
  • The paper provides an exhaustive set of experiments, including a complete set of ablation studies, demonstrating consistent performance improvements over text-based baselines across all evaluated configurations.
  • The paper provides clear full technical details of the proposed framework, from image rendering and the training procedure to the final retrieval setup.
Critical weaknesses
  • Limited Technical Novelty: The contribution is mainly an engineering work. The underlying architecture relies on standard RAG paradigms, standard indexing strategies, existing embedding models, and conventional contrastive training recipes without introducing novel methods.
  • MIssing comparison with more advanced baselines: 1) The main text only compares the method against basic text-based baselines, While additional evaluations with more advanced text-based retrieval methods are provided in the appendix, they should be extended and integrated and discussed into the main text. 2) Experiments comparing the proposed method with existing SoA visual retrievers are missing from the main text. Again the results included in the appendix should be integrated into the main text. The results included in the appendix show that the proposed approach actually performs worse than current SoA visual retrievers and this should be discussed in the main body of the paper.
  • In the evaluation on a corpus distinct from the training data (such as LiveQA), the base embedding model achieves better retrieval performance than the fine-tuned embedding model, suggesting poor generalization of the proposed framework.
Questions for authors

None

4 comments

W1. Technical novelty

Reviewer: "The contribution is mainly an engineering work ... without introducing novel methods."

Our key contribution is challenging a fundamental assumption in standard web RAG: web pages need to be parsed into text before retrieval. We demonstrate that web pages can instead be directly represented and retrieved as screenshots, eliminating the need for text parsing while preserving information that text extraction often loses. While our approach builds on standard RAG components, it fundamentally changes how web knowledge is represented and retrieved.

Our main finding is both surprising and practically important: visual retrieval outperforms strong text baselines even on well-established text-centric benchmarks such as NQ and SimpleQA. Prior visual RAG work, such as VisRAG and VDocRAG, primarily focuses on visually rich documents and visually oriented benchmarks, where visual information is central to the task and the advantage of pixels over OCR-based text is relatively intuitive. In contrast, web pages provide HTML from which text can be extracted directly, making the superiority of visual retrieval far from obvious. Beyond demonstrating this surprising result, we systematically investigate why pixels outperform text even on text-centric tasks.

Scale is also an important contribution in its own right. We evaluate visual retrieval across all 7.1M Wikipedia articles and 30M tiles, far beyond the thousands of documents typically used in prior visual RAG work. This scale enables new empirical findings in a realistic open-domain retrieval setting. We will also release the full index and processing pipeline, providing open-source infrastructure for future research on visual web retrieval.

Finally, we provide training and efficiency insights that make visual retrieval practical at this scale. Our training recipe addresses false negatives arising from repeated facts across web pages, improving validation accuracy from 0.725 to 0.793 (Table 4). We also demonstrate that pixel-based retrieval can reduce reader input tokens.

In short, our contribution is not a new RAG architecture, but establishing, explaining, and enabling visual retrieval at web scale, with empirical findings that prior visual document RAG work had not demonstrated.


W2. Advanced baselines and visual retrievers

Reviewer: "[Advanced text baselines and SoA visual retrievers] should be integrated into the main text ... The results included in the appendix show that the proposed approach actually performs worse than current SoA visual retrievers."

We agree that these comparisons deserve more visibility in the main paper. They are already included in Appendices C.3–C.5 and summarized in Sections 5.1 and 5.3, and we will move the detailed results into the main text in the revision. We believe this is primarily a presentation issue and does not change our paper’s conclusions.

Importantly, our claim is not that PixelRAG, with its current backbone, is the strongest visual retriever. Our central finding is that visual retrieval is an effective approach for web retrieval, even on traditionally text-centric benchmarks. Our training recipe is backbone-agnostic, and stronger visual encoders could further improve the approach.

The results support this claim. PixelRAG consistently outperforms all evaluated text baselines, including advanced parsers, learned extractors, hybrid retrieval, and text retrievers trained with our recipe (Tables 1, 10–12).


W3. Generalization on LiveVQA

Reviewer: "... the base embedding model achieves better retrieval performance than the fine-tuned embedding model, suggesting poor generalization of the proposed framework."

We agree that fine-tuning on Wikipedia alone does not necessarily improve performance on other domains. Our contribution is to demonstrate a label-free training recipe that can be applied to any target corpus, rather than claiming that a Wikipedia-trained model generalizes optimally to every domain. One can naturally extend this approach by adding training data from other domains, which we explicitly identify as future work (Section 5.2, Appendix E).

Importantly, even without any news-domain training, PixelRAG substantially outperforms text RAG on LiveVQA (70.0 vs. 59.0 accuracy; 33.3 vs. 16.2 R@3), demonstrating the effectiveness of visual retrieval beyond Wikipedia.

Dear Reviewer,

Thank you again for your thoughtful review and insightful questions. We have posted our rebuttal and would greatly appreciate your consideration of our responses as you reassess our work.

Please let us know if we can provide any additional information or clarification.

Best regards,
The Authors

Thank you for the detailed response. Extending visual retrieval to large-scale datasets such as wikipedia can be a valuable contribution, but then, the work should be clearly evaluated with respect to existing visual retrieval methods, and this is only partially done in the appendix. The proposed pipeline might have an advantage against other visual retrievers in terms of efficiency and computational cost when applied to large-scale datasets, but this is not shown and discussed quantitatively in the paper. I miss a clear comparison with SoA in visual retrieval, discussing the trade-off between retrieval performance, efficiency and computational cost, that could show the actual practical contribution of the proposed method.

Thank you for highlighting this trade-off. Appendix C.5, Table 13 compares ColPali v1.3 with PixelRAG on the same candidate pools. ColPali achieves higher R@1 and R@3, while both reach 98.8% R@10. However, extrapolating the measured per-tile storage to the full Wikipedia corpus gives a 7.9 TB fp16 index for ColPali versus 120 GB for PixelRAG.

To quantify search cost, we benchmarked exhaustive (brute-force) search, which ranks every tile, using the measured representation shapes (ColPali v1.3: 1,030×128 vectors per tile; PixelRAG: one 2,048-d vector per tile). On a single H100 (fp16, single query, top-10), ColPali MaxSim scoring takes 3.9 ms over 10k tiles and 36.9 ms over 100k tiles, versus 0.09 ms and 0.34 ms for PixelRAG (≈43–108× faster). On CPU over 10k tiles, it is 376 ms versus 3.6 ms (≈100×). ColPali's latency grows linearly with corpus size (~0.37 ms per 1k tiles on GPU). Linearly extrapolated to the full 30M-tile Wikipedia corpus, an exhaustive ColPali scan would require roughly 11 s of single-H100 compute per query, and its 7.9 TB index would need about 100 H100s just to hold in memory. By comparison, PixelRAG's full-corpus retrieval takes 14.0 ms over its 120 GB IVF-index (see our response to W3/Q4). Practical ColPali deployments at this scale would need approximate multi-vector indexing (e.g., PLAID or token pooling), which adds engineering complexity and can reduce accuracy.

MethodDevice10k tiles: index / latency100k tiles: index / latencyFull corpus (30M tiles): index / latency
ColPali v1.3H1002.6 GB / 3.9 ms26.4 GB / 36.9 ms7.9 TB / ~11 s (extrapolated)
PixelRAGH10041 MB / 0.09 ms0.41 GB / 0.34 ms120 GB / 14.0 ms (measured)

We agree that this quality–efficiency–cost trade-off deserves clearer discussion. We will move the comparison into the main text as a single table that puts retrieval quality (R@1/3/10), index size, and search latency side by side.

We would greatly appreciate your consideration of our responses as you reassess our work.

Comments on the paper

1 comment
Area_Chair_TcNvOpenReview ↗

Post-Rebuttal Discussion

The authors have provided detailed responses to the reviews. Please take another look at the rebuttal and discuss whether the main concerns have been adequately addressed, particularly regarding technical novelty, comparisons with prior visual RAG methods, and the strength of the empirical evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.