acceptodds
Under review as a conference paper at ICLR 2027

RAG-Anything: All-in-One RAG System

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a fundamental paradigm for expanding Large Language Models beyond their static training limitations. However, a critical misalignment exists between current RAG capabilities and real-world information environments. Modern knowledge repositories are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, and mathematical expressions. Yet existing RAG frameworks are limited to textual content, creating fundamental gaps when processing multimodal documents. We present RAG-Anything, a unified framework that enables comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual-graph construction to capture both cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval that combines structural knowledge navigation with semantic matching. This enables effective reasoning over heterogeneous content where relevant evidence spans multiple modalities. RAG-Anything demonstrates superior performance on challenging multimodal benchmarks, achieving significant improvements over state-of-the-art methods. Performance gains become particularly pronounced on long documents where traditional approaches fail. Our framework establishes a new paradigm for multimodal knowledge access, eliminating the architectural fragmentation that constrains current systems. Our framework is open-sourced at: bluehttps://anonymous.4open.science/r/RAG-Anything-D22B/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, RAG, Agent, Multimodal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KFEJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on question answering over documents where useful information may appear in figures, tables, equations, or ordinary text. In RAG-Anything, these different elements are parsed and organized into two graphs. One captures connections involving non-text content, while the other is a knowledge graph built from the text.

Critical strengths
  1. Questions about real documents often require information from several places, including figures, tables, and surrounding text. I think it makes sense to consider these sources together rather than handle them independently. The paper puts parsing, graph construction, retrieval, and the recovery of visual content into one pipeline. While the individual components are not entirely new, bringing them together in this setting is useful.
  2. The reported results are fairly strong. RAG-Anything reaches 63.4% on DocBench and 42.8% on MMLongBench, outperforming the baselines shown in Tables 2 and 3. I also found the breakdown by document length helpful, since long-document retrieval is an important part of the paper's motivation. The ablation results suggest that the full pipeline is beneficial, although they do not tell us exactly which parts of the proposed design account for the improvement.
Critical weaknesses
  1. One of the paper's main arguments is that converting multimodal documents into text can lose important information. However, my understanding of Sections 2.1.1 and 3.1 is that text descriptions, entity summaries, and a text embedding model still play a major role in the indexing process. This raises a question about what happens to visual details that are not captured in those descriptions. The original image may be available to the VLM when generating an answer, but the system still needs to retrieve the correct image in the first place.I was also unsure about how some of the structural relations are constructed. Appendix A.2 mentions relations such as row-of, column-of, and header-applies-to. In the main method, however, I could mainly identify relations extracted from descriptions and the belongs_to links. It would help to explain where these finer relations come from and whether their correctness is checked. Similarly, I could not find a sufficiently precise description of the final fusion and ranking procedure in Section 2.2.These details matter because the paper makes fairly broad claims about preserving multimodal structure and addressing fragmentation across modalities. I do not think the current description fully supports those claims. The discussion of previous RAG methods as largely text-only could also be more careful, given the existing work on multimodal retrieval.
  2. My main concern with the experiments is the lack of a direct comparison between the proposed dual-graph design and a simpler graph-based alternative. The Chunk-only ablation removes graph construction altogether. It shows that adding graph information helps, but it does not show why two graphs are better than one. For example, would a single graph built from the same parsed content and descriptions perform similarly? The w/o Reranker experiment does not answer this question either. A comparison that keeps the parser, retrieval settings, and context budget fixed while changing only the graph design would be much more informative.There is also an issue with the direct GPT-4o-mini baseline. Section 3.1 states that it is limited to 50 pages, while an important part of the paper focuses on much longer documents. I would like to know how those 50 pages are selected and whether the selection affects which evidence the baseline can access. Without this information, it is difficult to judge how much of the reported improvement comes from the proposed method rather than differences in the available context.I do not see the use of two benchmarks as a problem by itself. My concern is that the current experiments do not fully test the specific design choices that the paper emphasizes.
Questions for authors
  1. Section 3.1 mentions the use of text-embedding-3-large, but I am not sure how figures, tables, and equations are represented in the index. Does the model simply embed their text descriptions? Or are visual features also included? It would be helpful to see an example of what the parser produces for a figure or table, and what is actually stored in the index afterward.
  2. I had trouble understanding where some of the relations in Appendix A.2 come from. For tables, does the parser directly provide relations like row-of, column-of, and header-applies-to? Or does an LLM work them out from the generated descriptions? I have a similar question about the links between figure panels, captions, and axes. Another thing I could not follow was the ranking process in Section 2.2. How do the graph scores, embedding similarities, modality preferences, and reranker outputs actually affect which results are returned first?
  3. Could the authors include a comparison against a single-graph version, or a version without the multimodal anchor and cross-modal links, while keeping the rest of the pipeline unchanged? This would help establish what the dual-graph design contributes. Also, for the direct GPT-4o-mini baseline, how are the 50 input pages chosen, and what effect might this limit have on the long-document results?
4 comments

We agree that the 50-page cap can affect which evidence the direct GPT-4o-mini reader sees. Figure 2 also compares RAG-Anything with MMGraphRAG, where both systems could search the entire document.

DocBench document length (Fig. 2)RAG-Anything accuracyMMGraphRAG accuracyDifference
101–200 pages68.2%54.6%+13.6 pp
200+ pages68.8%55.0%+13.8 pp

For direct GPT-4o-mini, the original evaluation used up to 50 pages at 144 dpi. To measure how much the selection rule matters, we took all 91 answerable MMLongBench-Doc questions with valid evidence-page annotations in documents over 100 pages. The first 50 pages contained all annotated evidence pages for 37/91 questions. Taking 50 evenly spaced pages did so for 20/91. Ranking the pages by BM25 against the question and taking the top 50 raised that count to 65/91.

What goes into the index? The 3,072-dimensional vectors encode the text of chunks, entities and relations using text-embedding-3-large. For a figure, the encoder receives the caption and a description generated from its pixels. When its chunk is retrieved, the reader can also see the original crop. For a table, the chunk includes its parsed HTML, caption, footnotes and description. For an equation, it includes the parsed formula and its description.

Where do the relations come from? MinerU records table rows, headers and cell spans in the HTML. The graph builder explicitly links each entity extracted from an item's description to the item itself. Other semantic links, including a link between a panel and an axis, depend on what the model extracts from the description.

For example, DocBench asks which director attended the fewest board meetings in the Asaplus Resources annual report (DocBench document 85, Annual Report 2020). On PDF page 6, MinerU retained the separate cells for meetings held and attended. ONG Yih Ching's row reads 4 held, 3 attended. Every other director attended four of four. Retrieval returned both the link from ONG to this table and the extracted statement that he attended three of four meetings. The attendance table was the first crop the reader received. It named ONG and gave the correct count.

How are results ranked? First, we search the entity index with low-level query keywords and the relation index with high-level keywords, using vector similarity. In the entity branch, we rank neighboring edges by the sum of their endpoints' degrees, then by edge weight. We interleave and deduplicate the graph records from the two searches. For the chunk list, we take one direct dense match, then one entity-linked chunk, then one relation-linked chunk, and repeat. Finally, BGE reranks the deduplicated chunks before the count and token cutoffs. Entities and relations have their own token budgets. A query word such as “table” can influence the embedding or extracted keywords. The implementation has no separate modality weight.

On the same-content single graph. We removed the item anchors and membership edges while keeping the parser outputs, descriptions, dense chunks, reader and context limits fixed. Across 50 paired questions from seven MMLongBench-Doc documents, RAG-Anything answered 28/50 correctly and the single graph 26/50. On the 20 questions whose official evidence spans multiple pages, the counts were 8/20 and 7/20. We also inspected the reader inputs for 40 paired questions from three DocBench and four MMLongBench-Doc documents. The crop lists matched in content and order throughout. On 31/40 questions, only the full model supplied an explicit link from an extracted entity to the figure, table or equation it came from.

Anonymous supplementary materials: page selections, graph results and source documents

https://anonymous.4open.science/r/document-qa-review-8D42/KFEJ/README.md

This folder contains the selected pages and annotated evidence pages for each of the 91 questions in the 50-page comparison. It also includes the single-graph control and follows one DocBench table from parsing to the reader’s answer. The board attendance table is on PDF page 6 of Asaplus Resources Limited, Annual Report 2020 (DocBench 85).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Evidence

Dear Reviewer KFEJ,

Thank you again for your constructive feedback. As the rebuttal period is approaching its deadline, we would like to kindly follow up on our response.

We have provided additional clarifications and experiments addressing your main concerns, including: (1) how multimodal content is represented and linked in the index; (2) the detailed retrieval, fusion, and ranking procedure; (3) a controlled comparison with the same-content single-graph variant; and (4) an analysis of how the 50-page selection strategy affects evidence coverage. We have also released the corresponding question-level results and supporting materials for verification.

We hope these additions have addressed your concerns. If so, we would be very grateful if you could reconsider your assessment and score. If any questions remain, we would be more than happy to provide further clarification or additional evidence before the discussion closes.

Thank you very much for your time and consideration.

Dear Reviewer KFEJ,

Thank you again for your time and constructive feedback. As the discussion deadline is approaching, we wanted to kindly follow up on our response and the additional evidence we provided.

We would greatly appreciate it if you could take a moment to review the clarifications and let us know whether they address your concerns. If any questions remain, we would be happy to clarify them before the discussion closes.

Thank you very much for your consideration!

Best regards, The authors

Reviewer_crVHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAG-Anything is a framework for question answering over documents containing text, figures, tables, and equations. It uses VLM-generated descriptions to build a cross-modal graph, merges it with a text-based graph, and combines graph retrieval with vector search. Retrieved images are also passed to the VLM when generating answers. The paper reports improvements over GPT-4o-mini, LightRAG, and MMGraphRAG on DocBench and MMLongBench, particularly for long documents.

Critical strengths
  1. The appendix includes both successful and failed examples, which helps readers understand where the method works and where it still struggles.
Critical weaknesses
  1. The novelty seems limited. Converting multimodal content into textual descriptions and then combining graph construction, vector retrieval, and reranking is a familiar approach. The framework may be useful as a system, but I am less clear about the methodological contribution beyond putting these components together. A closer comparison with MMGraphRAG and MegaRAG would help explain what is new.

  2. An important gap is the lack of comparisons with 2026 methods. The paper discusses MegaRAG and UniversalRAG as 2026 work, but the experiments only compare against GPT-4o-mini, LightRAG, and MMGraphRAG. Comparisons with recent methods, especially the closely related MegaRAG, are needed to show whether the proposed design offers an advantage over current approaches.

Questions for authors

Please see the weaknesses above. I also have the following concerns and questions.

  1. The cross-modal graph and its retrieval representations depend on descriptions generated by a VLM. If a description misreads a chart, confuses table headers, or does not match the meaning of the original image, these errors may enter the graph and affect retrieval. I understand that the original images are recovered during generation, but that may not help if the correct evidence was not retrieved in the first place. How does the method detect or handle this mismatch?

  2. The ablation study does not directly test the main dual-graph design. Table 4 only removes the graph entirely or removes the reranker. This shows that the graph helps, but it does not show why two graphs are needed. Could the authors compare the dual-graph design with each graph alone and with a single graph built from the same content?

  3. The cost of building and using the index is unclear. Processing non-text content with a VLM and constructing the graph could be expensive for long documents. Could the authors report how long indexing takes and measure query latency?

  4. All answers are judged by GPT-4o-mini, which is also the backbone used in the experiments. Are there other ways to evaluate answer correctness? Also, OpenAI has announced that GPT-4o-mini will be removed from the API on January 14, 2027. Could the authors use another judge model so that the evaluation can still be reproduced after that date?

9 comments

We agree that Table 4 leaves open what the second graph adds. We ran the single-graph control you suggested. Both versions index the same table description. In the full model, the table also has its own node, and each entity extracted from the description has a belongs_to edge to that node. Matching entity names connect those records to mentions in the prose. The single graph keeps the descriptions and omits the item nodes and membership edges.

On the graph control. We held the parser outputs, descriptions, dense chunks, reader and context limits fixed across 40 paired questions from three DocBench and four MMLongBench-Doc documents.

Graph used for retrievalCorrect answers
RAG-Anything (both branches with item anchors)29/40
Single graph built from the same content28/40
Text branch alone28/40
Multimodal item branch alone27/40

We also checked what the reader received. The full model and single graph supplied identical ordered lists of crops on all 40 questions. On 31/40, only the full model supplied an explicit link from an extracted entity to the item it came from.

On the comparison with other graph methods. MMGraphRAG uses spectral clustering to align entities across modalities. Our membership links let the reader trace an entity back to a specific figure, table or equation. On the multimodal questions in DocBench (Table 2), RAG-Anything scored 76.3% and MMGraphRAG 66.0%. We prioritized MegaRAG for the new experiment because, like RAG-Anything, it builds a multimodal graph over the document supplied with each question. UniversalRAG also addresses routing across corpora. We ran adapted MegaRAG with its GME encoder, shared MinerU input and the same Qwen3.8-Flash reader. On the 14 MMLongBench-Doc cross-page questions with graded answers from both systems, each answered eight correctly.

On indexing time and query latency. Across the seven documents used for the graph control, the measured indexing stages totaled 4.88 hours for RAG-Anything and 9.19 hours for adapted MegaRAG. RAG-Anything took less indexing time on each document. Once the index was built, median query latency for RAG-Anything on the 112-page Wikidata map-making workshop slides (MMLongBench-Doc) was 49.1 seconds.

On errors in generated descriptions. We share your concern that a reader can inspect only what retrieval brings back. For a table, we index the parsed body and caption alongside the generated description and retain the original crop for inspection. Semantic errors in a description are not automatically flagged during indexing. To check fidelity against the source, we selected one field in each of two figures and four tables from four documents, two per benchmark, before examining their descriptions. All six fields were preserved. For example, the description of the chart on PDF page 6 of Pew Research Center's Women and Men in STEM Often at Odds Over Workplace Equity (MMLongBench-Doc) correctly assigned the 19% figure to men in STEM who had experienced gender discrimination at work.

On independent grading. Qwen3.8-Flash generated the new answers. Doubao Seed 2.1 Pro graded them with method names masked and passed all 12 rubric checks. We also used the open-weight Ministral-3-8B-Instruct-2512 to rescore 20 matched questions on three annual reports, Nike and Adobe from MMLongBench-Doc and Texas Instruments from DocBench.

JudgeRAG-AnythingSame-content single graphColPali
Doubao9/209/207/20
Ministral8/208/206/20

Both judges agreed that RAG-Anything answered the Nike receivables and Adobe cash-flow questions correctly where ColPali did not.

Anonymous supplementary materials: results and source documents

https://anonymous.4open.science/r/document-qa-review-8D42/crVH/README.md

This folder contains the question-level results for the graph ablations and the comparison with adapted MegaRAG, including indexing times for each document. It also gives both judges’ grades and identifies the original PDF page for each description we checked. For example, the 19% figure discussed in our reply appears on PDF page 6 of Pew Research Center’s report on workplace equity in STEM.

I thank the authors for their response and the effort put into this work. However, I feel that my concerns have not yet been fully addressed.

First, regarding incorrect descriptions caused by hallucinations, checking only six fields from two figures and four tables seems insufficient to establish statistical significance. To my knowledge, such hallucinations are widespread. Otherwise, performance on any multimodal benchmark would theoretically reach 100%. I do not think that simply retaining the original images mitigates this issue.

Second, the speed comparison with MegaRAG alone does not address my concern about scalability. The results do not show how processing time changes as more data is added. In my experience, the cost of building a graph can grow quadratically rather than linearly. Converting images and tables into text descriptions adds further overhead.

I appreciate the authors’ effort, but these concerns remain, and my assessment is unchanged.

Reviewer_fRXnOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates retrieval-augmented generation (RAG) for multimodal documents containing text, images, tables, and equations. Existing RAG methods often process these elements separately or convert them into text, potentially losing important structural information. To address this limitation, the authors propose RAG-Anything, a unified framework that uses two knowledge graphs to represent textual and multimodal information. The framework combines graph-based retrieval with semantic similarity matching to identify relevant information for question answering, without requiring additional model training. Experiments on DocBench and MMLongBench show that RAG-Anything outperforms the compared baselines, particularly when handling long documents.

Critical strengths
  1. This paper addresses a practical problem in multimodal RAG. I think it is useful to represent text, images, tables, and equations within a unified framework, as these elements are often closely related in real-world documents. The proposed dual-graph design provides a reasonable way to preserve these relationships and support retrieval across different modalities.

  2. The paper evaluates RAG-Anything on two multimodal document benchmarks covering different document types and domains. The results show that it outperforms the baseline methods, particularly on long documents. The ablation study further demonstrates that graph-based retrieval improves performance compared to a chunk-only approach. These findings provide evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. Some relevant multimodal retrieval approaches, such as VisRAG, ColPali, and VDocRAG, are discussed in the related work but are not included in the experiments.

  2. The paper does not report detailed indexing costs or inference efficiency.

Questions for authors
  1. Could the authors provide comparison with more multimodal retrieval approaches, or explain why did not include these methods?

  2. Could the authors provide more details about the inference efficiency.

6 comments

We thank the reviewer for suggesting VisRAG and ColPali. We compared all three methods with the same Qwen3.8-Flash reader on seven MMLongBench-Doc documents. The table gives the number of correct answers. The second row is the subset whose official evidence spans multiple pages, identified before grading.

QuestionsRAG-AnythingVisRAGColPali
All matched questions28/5026/5024/50
Cross-page subset8/204/205/20

The comparison includes all 50 questions in our VisRAG evaluation set with answers from both RAG-Anything and ColPali. The documents comprise two company filings, three Pew Research Center reports and two sets of teaching slides. Each method searched the whole document. VisRAG and ColPali gave the reader their five highest-ranked pages. RAG-Anything gave it retrieved text, graph relations and crops from the source. Doubao Seed 2.1 Pro graded the answers with method names masked. On five cross-page questions, RAG-Anything gave the correct answer and VisRAG missed it. The reverse occurred once.

One MMLongBench-Doc question asks for receivables turnover in Nike’s FY2021 Form 10-K. The first two crops sent to our reader came from tables on PDF pages 91 and 92. They contain net revenue of 44,538 and receivables of 4,463 and 2,749 for the two fiscal year ends (all in millions of dollars). The reader averaged the two balances and divided revenue by 3,606 to obtain 12.35. Given the five pages selected by either VisRAG or ColPali, the same reader said it lacked the figures to compute the ratio.

On indexing and query cost. For the timing comparison with MegaRAG, we used the three DocBench and four MMLongBench-Doc documents with complete indexing times for both systems. Summing the recorded indexing stages gave 4.88 hours for RAG-Anything and 9.19 hours for adapted MegaRAG. RAG-Anything took less time on each document. On the 112-page Wikidata map-making workshop slides in MMLongBench-Doc, median latency across six questions was 49.1 seconds. Total metered API cost was at most US$$ 0.88 to index the slides and about US$ 0.025 for the six queries, excluding hardware.

How much visual input reaches the reader? For this measurement we used a broader set of 60 matched questions from ten documents (four DocBench, six MMLongBench-Doc). RAG-Anything sent 64.9% fewer image pixels than ColPali and answered 38 questions correctly versus 36. On the 43 questions for which both methods sent exactly five images, sending cropped items instead of whole pages still reduced the pixel count by 52.9%.

Thank you for the additional experiments. The new comparisons with VisRAG and ColPali help address my concerns regarding multimodal retrieval baselines, and the reported indexing time and API costs provide useful additional insights.

Two aspects remain insufficiently addressed. First, VDocRAG, which was also mentioned in my original review, is still not included in the comparison, and could the authors explain its omission? Second, query latency is reported only for six questions on a single document, without comparisons against other methods.

Overall, I appreciate the additional efforts, which have improved the empirical support for the proposed framework.

Thank you for following up on VDocRAG and query latency.

VDocRAG. In the first round we chose VisRAG and ColPali so that we could compare page retrieval while keeping the answer model fixed. Swapping out VDocRAG’s trained generator would test its retriever alone. We have now evaluated both released components together on the same 50 questions from seven MMLongBench-Doc PDFs. For each question, VDocRetriever ranked every page of its source PDF, and VDocGenerator answered from the top three. The other three systems used Qwen3.8-Flash, so the VDocRAG column compares complete systems, including their readers.

We used the authors’ released checkpoints with the following configuration, without fine-tuning on this evaluation set.

ItemVDocRAG configuration
ModelsPhi-3-vision-128k-instruct with the released VDocRetriever-Phi3-vision and VDocGenerator-Phi3-vision adapters
PagesAll 593 pages rendered at 144 dpi; 1344 × 1344 RGB resize from the official quick start; base processor with num_crops=16
RetrievalNormalized EOS embeddings; top 3 pages from each question’s complete PDF, following the released generation command
GenerationOfficial chat template and Answer briefly. prompt; greedy decoding with the released 64-token limit

We reproduced both official example retrieval rankings and the example answer, 28.69m. The run used BF16 on CPU with eager or query-row-blocked eager attention. The two implementations also returned identical text on a three-image control and the Adobe FY2015 Form 10-K question with the same retrieved pages. All 50 questions returned answers and were graded. The supplementary configuration record provides pinned versions, inference code, these checks and each question’s selected pages and output.

QuestionsRAG-AnythingVisRAGColPaliVDocRAG
All 5028/5026/5024/508/50
Cross-page (20 of 50)8/204/205/203/20

RAG-Anything answered 20 more questions correctly than VDocRAG on this set. On the 20 questions whose annotated evidence spans multiple pages, it answered five more correctly. We used the same cross-page subset and the same judge (Doubao Seed 2.1 Pro) as before, with method names hidden during grading.

For the FY2021 receivables-turnover question in Nike’s Form 10-K, VDocRAG selected PDF pages 102, 2 and 12 and answered 10.00. RAG-Anything retrieved table crops on pages 91 and 92 containing both year-end balances and net revenue, and obtained the reference answer of 12.35.

Query latency. We have now compared all three systems on the same 40 questions from our seven-document indexing study (three DocBench and four MMLongBench-Doc PDFs, 12–128 pages). All three methods used Qwen3.8-Flash. These times include retrieval and answer generation after the index was built.

Post-index query latency (seconds)MeanMedian95th percentile
RAG-Anything39.138.262.5
ColPali8.87.914.9
Adapted MegaRAG27.227.938.3

RAG-Anything had the longest median query time. It indexed each of the seven PDFs faster than adapted MegaRAG, taking 4.88 hours in total compared with 9.19 hours. Thus the time saving we observed was in indexing.

Comments on the paper

1 comment
Area_Chair_3ZUqOpenReview ↗

Please engage with the authors rebuttal by Nov 18

Dear Reviewers,

The authors have posted their rebuttal. Please engage with it by November 18, 2026, noting which of your concerns are resolved or which remain. Even brief engagement helps greatly.

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.