- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MAGE-RAG, an agentic RAG pipeline for long-document multimodal QA. Offline, each PDF is parsed into a graph with page nodes and element nodes (paragraphs, tables, figures, etc.), connected by containment, reading-order, layout, section and semantic-similarity edges. Online, ColPali retrieves the top-k pages, and then an LVLM "controller" (Qwen3-VL-8B) runs for a few rounds. In each round it can activate neighboring nodes, open elements, search for more pages, or prune evidence, under limits on k, the number of rounds T, and the selections per round a. The retained evidence is rendered as page-organized text plus images and passed to a reader (Qwen3-VL-8B).
The paper reports 52.75 accuracy on LongDocURL and 53.26 accuracy / 51.19 F1 on MMLongBench-Doc. These numbers are higher than the reproduced baselines (Direct MLLM, BM25, ColBERTv2, M3DocRAG, EVisRAG, G2-Reader) and one number copied from the MLDocRAG paper. Validation consists of the main tables, subgroup breakdowns, budget sweeps (k, T, a), ablations on the controller and graph relations (MMLongBench-Doc only), and trace/logical-cost statistics from one run.
Critical strengths
-
Relevant problem and a consistent gain over one-shot visual RAG on cross-page questions. Under the same Qwen3-VL-8B reader, MAGE-RAG reaches 34.70 on MMLongBench-Doc cross-page questions, versus 24.87 for M3DocRAG and 24.51 for EVisRAG, and it is also above G2-Reader (30.69). Cross-page evidence is exactly where fixed top-k retrieval is expected to fail, so this is the most convincing part of the empirical story.
-
Transparency and reproducibility. The appendix describes the graph construction, the controller interface, the stopping rules and the budget accounting in unusual detail, and code and configs are provided. The authors also openly say that the budget curves are non-monotonic, that trace statistics do not separate correct from incorrect answers, and that their logical-cost numbers do not establish efficiency. This honesty made it possible for me to check the claims against the tables, and I appreciate it.
Critical weaknesses
-
The experiments do not support the claimed contribution of the individual components, and the key ablation is missing its reference point. Table 3 fixes (k,T,a)=(3,10,5), but no full model at this configuration is reported anywhere. The text instead compares the no-search variant (22.15 cross-page, k=3) with 34.70 "for the full configuration in the main results", which is the k=5 run from Table 2(d). The two are not comparable. The numbers that can be compared are not encouraging. Several ablated variants reach about 52.8-53.0 accuracy (No Prune 52.99, Semantic Graph 52.96, Graph Neighbor Expansion 52.88, Page Only 52.84), which is the same as, or higher than, the full model at k=3 in Table 2 (about 52.0-53.1 depending on T and a). Adding more relation types does not help either: the full Structural Graph (50.05) is worse than Containment Only (51.21) and Page Only (52.84). The paper itself says the results "do not support attributing the full result to a single relation type", yet the abstract says the ablations "support the value of organizing cross-page and heterogeneous evidence". In addition, everything is a single run on about 1,091 questions, so the 95% CI is roughly +-3 points and most differences discussed (1-2 points) are within noise. Ablations and budget sweeps are only done on MMLongBench-Doc, not LongDocURL. Without a same-configuration full model, repeated runs, and this analysis on both benchmarks, I cannot tell which of the many design elements (multigranular graph, five relation types, search, prune, controller) actually matter.
-
The need for a complex LLM controller is not demonstrated, and the headline gain is mostly not where the paper says it is. Going from no controller (T=0, 50.01) to T=5 (52.25) gives about 2 points, it is not monotonic (T=1 is lower than T=0), and a simple graph-neighbor expansion without the full controller already reaches 52.88. On MMLongBench-Doc, Direct MLLM actually has a higher cross-page score (38.58) than MAGE-RAG (34.70). The overall improvement over Direct MLLM (43.05 to 53.26) comes mainly from the unanswerable subset (27.62 to 68.03). By my rough calculation, using the benchmark's approximate subset sizes (about 45% single-page, 29% cross-page, 26% unanswerable), this accounts for almost all of the roughly 10-point gain. This looks like an abstention-behavior effect (the reader prompt explicitly asks for "Not answerable." when evidence is thin), not better evidence construction, and the paper's own statement that the advantage lies in "answerable questions requiring visual or cross-page evidence" is not supported by these numbers. On LongDocURL the margin over Direct MLLM is 2.7 points, and the multi-page score (47.69) is below Direct MLLM (48.44). The final setting (k=5, T=10, a=5) appears to be the best point of a sweep run on the evaluation set itself, while baselines are not tuned in the same way. There are also no simple strong baselines, such as a larger top-k with a matched image budget, a heuristic non-LLM expansion, or a few-sample voting baseline, to show that an LLM controller is needed.
-
Execution budget is not resource efficiency, and there is no fair cost-versus-accuracy comparison. The budgets only bound k, the number of rounds, and the number of activation selections (and the latter can overshoot, up to Amax + a - 1). Open, search and prune actions are not counted. The two case studies in Appendix F show about 100 opened nodes, 10 searches and 105 reader images per question, and in the first case 10 pages are activated, which seems to be nearly the whole document. Both runs also end through the watchdog instead of a voluntary stop. The averages in Table 6 (about 21 images and 3.2 LLM calls per question) are only reported for MAGE-RAG, with no matching numbers for Direct MLLM, M3DocRAG, EVisRAG or G2-Reader, which is itself an agentic method. The Score/Image and Score/LLM numbers in Table 6 are split by correct vs. incorrect samples, which makes them circular and not informative about efficiency. The authors acknowledge that cost superiority is not established and that latency is not measured. Since the central motivation of the paper is solving the "coverage-noise-cost trade-off" within a finite budget, a missing cost comparison leaves the main claim untested. In addition, the MLDocRAG row is copied from another paper with a different reader, and VisRAG is listed as a baseline but does not appear in Table 1.
Questions for authors
- Please report the full model at exactly the Table 3 setting (k=3, T=10, a=5), from the same run batch as the ablations ("reported batch" is mentioned in the text), with several seeds or bootstrap confidence intervals. Please also explain why the no-search result (k=3) is compared with the 34.70 result (k=5). In addition, in Table 2(a) the k=10 row has 50.10 accuracy but 58.61 F1, while F1 is below accuracy in every other row. Is this a typo or a real number? (This is a clarification request that affects how I read the budget analysis.)
Hi, authors. The reviewer-author discussion phase is comming to an end, I havenot yet received any response for my comments, so my concerns remain.
If you have new results or explanations, please respond asap.
Thanks.