- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on question answering over documents where useful information may appear in figures, tables, equations, or ordinary text. In RAG-Anything, these different elements are parsed and organized into two graphs. One captures connections involving non-text content, while the other is a knowledge graph built from the text.
Critical strengths
- Questions about real documents often require information from several places, including figures, tables, and surrounding text. I think it makes sense to consider these sources together rather than handle them independently. The paper puts parsing, graph construction, retrieval, and the recovery of visual content into one pipeline. While the individual components are not entirely new, bringing them together in this setting is useful.
- The reported results are fairly strong. RAG-Anything reaches 63.4% on DocBench and 42.8% on MMLongBench, outperforming the baselines shown in Tables 2 and 3. I also found the breakdown by document length helpful, since long-document retrieval is an important part of the paper's motivation. The ablation results suggest that the full pipeline is beneficial, although they do not tell us exactly which parts of the proposed design account for the improvement.
Critical weaknesses
- One of the paper's main arguments is that converting multimodal documents into text can lose important information. However, my understanding of Sections 2.1.1 and 3.1 is that text descriptions, entity summaries, and a text embedding model still play a major role in the indexing process. This raises a question about what happens to visual details that are not captured in those descriptions. The original image may be available to the VLM when generating an answer, but the system still needs to retrieve the correct image in the first place.I was also unsure about how some of the structural relations are constructed. Appendix A.2 mentions relations such as row-of, column-of, and header-applies-to. In the main method, however, I could mainly identify relations extracted from descriptions and the belongs_to links. It would help to explain where these finer relations come from and whether their correctness is checked. Similarly, I could not find a sufficiently precise description of the final fusion and ranking procedure in Section 2.2.These details matter because the paper makes fairly broad claims about preserving multimodal structure and addressing fragmentation across modalities. I do not think the current description fully supports those claims. The discussion of previous RAG methods as largely text-only could also be more careful, given the existing work on multimodal retrieval.
- My main concern with the experiments is the lack of a direct comparison between the proposed dual-graph design and a simpler graph-based alternative. The Chunk-only ablation removes graph construction altogether. It shows that adding graph information helps, but it does not show why two graphs are better than one. For example, would a single graph built from the same parsed content and descriptions perform similarly? The w/o Reranker experiment does not answer this question either. A comparison that keeps the parser, retrieval settings, and context budget fixed while changing only the graph design would be much more informative.There is also an issue with the direct GPT-4o-mini baseline. Section 3.1 states that it is limited to 50 pages, while an important part of the paper focuses on much longer documents. I would like to know how those 50 pages are selected and whether the selection affects which evidence the baseline can access. Without this information, it is difficult to judge how much of the reported improvement comes from the proposed method rather than differences in the available context.I do not see the use of two benchmarks as a problem by itself. My concern is that the current experiments do not fully test the specific design choices that the paper emphasizes.
Questions for authors
- Section 3.1 mentions the use of text-embedding-3-large, but I am not sure how figures, tables, and equations are represented in the index. Does the model simply embed their text descriptions? Or are visual features also included? It would be helpful to see an example of what the parser produces for a figure or table, and what is actually stored in the index afterward.
- I had trouble understanding where some of the relations in Appendix A.2 come from. For tables, does the parser directly provide relations like row-of, column-of, and header-applies-to? Or does an LLM work them out from the generated descriptions? I have a similar question about the links between figure panels, captions, and axes. Another thing I could not follow was the ranking process in Section 2.2. How do the graph scores, embedding similarities, modality preferences, and reranker outputs actually affect which results are returned first?
- Could the authors include a comparison against a single-graph version, or a version without the multimodal anchor and cross-modal links, while keeping the rest of the pipeline unchanged? This would help establish what the dual-graph design contributes. Also, for the direct GPT-4o-mini baseline, how are the 50 input pages chosen, and what effect might this limit have on the long-document results?
4 comments
Anonymous supplementary materials: page selections, graph results and source documents
https://anonymous.4open.science/r/document-qa-review-8D42/KFEJ/README.md
This folder contains the selected pages and annotated evidence pages for each of the 91 questions in the 50-page comparison. It also includes the single-graph control and follows one DocBench table from parsing to the reader’s answer. The board attendance table is on PDF page 6 of Asaplus Resources Limited, Annual Report 2020 (DocBench 85).
Follow-up on Our Rebuttal and Additional Evidence
Dear Reviewer KFEJ,
Thank you again for your constructive feedback. As the rebuttal period is approaching its deadline, we would like to kindly follow up on our response.
We have provided additional clarifications and experiments addressing your main concerns, including: (1) how multimodal content is represented and linked in the index; (2) the detailed retrieval, fusion, and ranking procedure; (3) a controlled comparison with the same-content single-graph variant; and (4) an analysis of how the 50-page selection strategy affects evidence coverage. We have also released the corresponding question-level results and supporting materials for verification.
We hope these additions have addressed your concerns. If so, we would be very grateful if you could reconsider your assessment and score. If any questions remain, we would be more than happy to provide further clarification or additional evidence before the discussion closes.
Thank you very much for your time and consideration.
Dear Reviewer KFEJ,
Thank you again for your time and constructive feedback. As the discussion deadline is approaching, we wanted to kindly follow up on our response and the additional evidence we provided.
We would greatly appreciate it if you could take a moment to review the clarifications and let us know whether they address your concerns. If any questions remain, we would be happy to clarify them before the discussion closes.
Thank you very much for your consideration!
Best regards, The authors
We agree that the 50-page cap can affect which evidence the direct GPT-4o-mini reader sees. Figure 2 also compares RAG-Anything with MMGraphRAG, where both systems could search the entire document.
For direct GPT-4o-mini, the original evaluation used up to 50 pages at 144 dpi. To measure how much the selection rule matters, we took all 91 answerable MMLongBench-Doc questions with valid evidence-page annotations in documents over 100 pages. The first 50 pages contained all annotated evidence pages for 37/91 questions. Taking 50 evenly spaced pages did so for 20/91. Ranking the pages by BM25 against the question and taking the top 50 raised that count to 65/91.
What goes into the index? The 3,072-dimensional vectors encode the text of chunks, entities and relations using text-embedding-3-large. For a figure, the encoder receives the caption and a description generated from its pixels. When its chunk is retrieved, the reader can also see the original crop. For a table, the chunk includes its parsed HTML, caption, footnotes and description. For an equation, it includes the parsed formula and its description.
Where do the relations come from? MinerU records table rows, headers and cell spans in the HTML. The graph builder explicitly links each entity extracted from an item's description to the item itself. Other semantic links, including a link between a panel and an axis, depend on what the model extracts from the description.
For example, DocBench asks which director attended the fewest board meetings in the Asaplus Resources annual report (DocBench document 85, Annual Report 2020). On PDF page 6, MinerU retained the separate cells for meetings held and attended. ONG Yih Ching's row reads 4 held, 3 attended. Every other director attended four of four. Retrieval returned both the link from ONG to this table and the extracted statement that he attended three of four meetings. The attendance table was the first crop the reader received. It named ONG and gave the correct count.
How are results ranked? First, we search the entity index with low-level query keywords and the relation index with high-level keywords, using vector similarity. In the entity branch, we rank neighboring edges by the sum of their endpoints' degrees, then by edge weight. We interleave and deduplicate the graph records from the two searches. For the chunk list, we take one direct dense match, then one entity-linked chunk, then one relation-linked chunk, and repeat. Finally, BGE reranks the deduplicated chunks before the count and token cutoffs. Entities and relations have their own token budgets. A query word such as “table” can influence the embedding or extracted keywords. The implementation has no separate modality weight.
On the same-content single graph. We removed the item anchors and membership edges while keeping the parser outputs, descriptions, dense chunks, reader and context limits fixed. Across 50 paired questions from seven MMLongBench-Doc documents, RAG-Anything answered 28/50 correctly and the single graph 26/50. On the 20 questions whose official evidence spans multiple pages, the counts were 8/20 and 7/20. We also inspected the reader inputs for 40 paired questions from three DocBench and four MMLongBench-Doc documents. The crop lists matched in content and order throughout. On 31/40 questions, only the full model supplied an explicit link from an extracted entity to the figure, table or equation it came from.