acceptodds
Withdrawn from ICLR 2027

MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-document multimodal question answering requires evidence whose amount, location, and modality vary with the question. One-shot document RAG typically retrieves a fixed number of text chunks or page images, leaving limited scope to recover missing clues, inspect local details, or remove irrelevant content before answering. Constructing sufficient evidence within a finite budget therefore requires query-dependent decisions about both the extent and granularity of the reader context. We propose MAGE-RAG for budgeted query-time evidence subgraph construction. It connects pages and local elements in an offline graph, uses an online controller to select and revise evidence from retrieved entry pages, and renders the resulting state as page-organized multimodal input. MAGE-RAG achieves 52.75 accuracy on LongDocURL and 53.26 accuracy with 51.19 F1 on MMLongBench-Doc. Breakdown and ablation results support the value of organizing cross-page and heterogeneous evidence; budget experiments further show that increasing the initial page count or controller budget does not consistently improve accuracy. Our code is available at https://anonymous.4open.science/r/MAGE-RAG-F8B3.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal RAG, Long-document QA, Evidence graphs, Agentic retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_Fp9eOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAGE-RAG, an agentic RAG pipeline for long-document multimodal QA. Offline, each PDF is parsed into a graph with page nodes and element nodes (paragraphs, tables, figures, etc.), connected by containment, reading-order, layout, section and semantic-similarity edges. Online, ColPali retrieves the top-k pages, and then an LVLM "controller" (Qwen3-VL-8B) runs for a few rounds. In each round it can activate neighboring nodes, open elements, search for more pages, or prune evidence, under limits on k, the number of rounds T, and the selections per round a. The retained evidence is rendered as page-organized text plus images and passed to a reader (Qwen3-VL-8B).

The paper reports 52.75 accuracy on LongDocURL and 53.26 accuracy / 51.19 F1 on MMLongBench-Doc. These numbers are higher than the reproduced baselines (Direct MLLM, BM25, ColBERTv2, M3DocRAG, EVisRAG, G2-Reader) and one number copied from the MLDocRAG paper. Validation consists of the main tables, subgroup breakdowns, budget sweeps (k, T, a), ablations on the controller and graph relations (MMLongBench-Doc only), and trace/logical-cost statistics from one run.

Critical strengths
  1. Relevant problem and a consistent gain over one-shot visual RAG on cross-page questions. Under the same Qwen3-VL-8B reader, MAGE-RAG reaches 34.70 on MMLongBench-Doc cross-page questions, versus 24.87 for M3DocRAG and 24.51 for EVisRAG, and it is also above G2-Reader (30.69). Cross-page evidence is exactly where fixed top-k retrieval is expected to fail, so this is the most convincing part of the empirical story.

  2. Transparency and reproducibility. The appendix describes the graph construction, the controller interface, the stopping rules and the budget accounting in unusual detail, and code and configs are provided. The authors also openly say that the budget curves are non-monotonic, that trace statistics do not separate correct from incorrect answers, and that their logical-cost numbers do not establish efficiency. This honesty made it possible for me to check the claims against the tables, and I appreciate it.

Critical weaknesses
  1. The experiments do not support the claimed contribution of the individual components, and the key ablation is missing its reference point. Table 3 fixes (k,T,a)=(3,10,5), but no full model at this configuration is reported anywhere. The text instead compares the no-search variant (22.15 cross-page, k=3) with 34.70 "for the full configuration in the main results", which is the k=5 run from Table 2(d). The two are not comparable. The numbers that can be compared are not encouraging. Several ablated variants reach about 52.8-53.0 accuracy (No Prune 52.99, Semantic Graph 52.96, Graph Neighbor Expansion 52.88, Page Only 52.84), which is the same as, or higher than, the full model at k=3 in Table 2 (about 52.0-53.1 depending on T and a). Adding more relation types does not help either: the full Structural Graph (50.05) is worse than Containment Only (51.21) and Page Only (52.84). The paper itself says the results "do not support attributing the full result to a single relation type", yet the abstract says the ablations "support the value of organizing cross-page and heterogeneous evidence". In addition, everything is a single run on about 1,091 questions, so the 95% CI is roughly +-3 points and most differences discussed (1-2 points) are within noise. Ablations and budget sweeps are only done on MMLongBench-Doc, not LongDocURL. Without a same-configuration full model, repeated runs, and this analysis on both benchmarks, I cannot tell which of the many design elements (multigranular graph, five relation types, search, prune, controller) actually matter.

  2. The need for a complex LLM controller is not demonstrated, and the headline gain is mostly not where the paper says it is. Going from no controller (T=0, 50.01) to T=5 (52.25) gives about 2 points, it is not monotonic (T=1 is lower than T=0), and a simple graph-neighbor expansion without the full controller already reaches 52.88. On MMLongBench-Doc, Direct MLLM actually has a higher cross-page score (38.58) than MAGE-RAG (34.70). The overall improvement over Direct MLLM (43.05 to 53.26) comes mainly from the unanswerable subset (27.62 to 68.03). By my rough calculation, using the benchmark's approximate subset sizes (about 45% single-page, 29% cross-page, 26% unanswerable), this accounts for almost all of the roughly 10-point gain. This looks like an abstention-behavior effect (the reader prompt explicitly asks for "Not answerable." when evidence is thin), not better evidence construction, and the paper's own statement that the advantage lies in "answerable questions requiring visual or cross-page evidence" is not supported by these numbers. On LongDocURL the margin over Direct MLLM is 2.7 points, and the multi-page score (47.69) is below Direct MLLM (48.44). The final setting (k=5, T=10, a=5) appears to be the best point of a sweep run on the evaluation set itself, while baselines are not tuned in the same way. There are also no simple strong baselines, such as a larger top-k with a matched image budget, a heuristic non-LLM expansion, or a few-sample voting baseline, to show that an LLM controller is needed.

  3. Execution budget is not resource efficiency, and there is no fair cost-versus-accuracy comparison. The budgets only bound k, the number of rounds, and the number of activation selections (and the latter can overshoot, up to Amax + a - 1). Open, search and prune actions are not counted. The two case studies in Appendix F show about 100 opened nodes, 10 searches and 105 reader images per question, and in the first case 10 pages are activated, which seems to be nearly the whole document. Both runs also end through the watchdog instead of a voluntary stop. The averages in Table 6 (about 21 images and 3.2 LLM calls per question) are only reported for MAGE-RAG, with no matching numbers for Direct MLLM, M3DocRAG, EVisRAG or G2-Reader, which is itself an agentic method. The Score/Image and Score/LLM numbers in Table 6 are split by correct vs. incorrect samples, which makes them circular and not informative about efficiency. The authors acknowledge that cost superiority is not established and that latency is not measured. Since the central motivation of the paper is solving the "coverage-noise-cost trade-off" within a finite budget, a missing cost comparison leaves the main claim untested. In addition, the MLDocRAG row is copied from another paper with a different reader, and VisRAG is listed as a baseline but does not appear in Table 1.

Questions for authors
  1. Please report the full model at exactly the Table 3 setting (k=3, T=10, a=5), from the same run batch as the ablations ("reported batch" is mentioned in the text), with several seeds or bootstrap confidence intervals. Please also explain why the no-search result (k=3) is compared with the 34.70 result (k=5). In addition, in Table 2(a) the k=10 row has 50.10 accuracy but 58.61 F1, while F1 is below accuracy in every other row. Is this a typo or a real number? (This is a clarification request that affects how I read the budget analysis.)
1 comment

Hi, authors. The reviewer-author discussion phase is comming to an end, I havenot yet received any response for my comments, so my concerns remain.

If you have new results or explanations, please respond asap.

Thanks.

Reviewer_dtK4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAGE-RAG, a graph-based agentic RAG framework for long-document multimodal QA. It builds a multigranular graph of pages and local elements, then uses a query-time controller to activate, inspect, search, and prune evidence before rendering the selected content for a multimodal reader. The method is evaluated on LongDocURL and MMLongBench-Doc and reports strong overall results, especially on cross-page questions. The main idea of adaptive multigranular evidence construction is reasonable and practically useful, but the experimental protocol has several weaknesses, particularly in hyperparameter selection, ablation design, and computational-cost evaluation, which make the reported gains less convincing.

Critical strengths

· The evidence-state interface is concrete and inspectable. The Inactive/Active/Opened/Pruned states and explicit Activate/Open/Search/Prune operations make the query-time behavior easier to trace than in a conventional one-shot RAG pipeline. · The multigranular representation is sensible for visually rich long documents. Page nodes preserve layout and surrounding context, while local elements provide more focused evidence.

Critical weaknesses

· The paper performs extensive sweeps over the initial page count kk, controller rounds TT, and actions per round aa directly on MMLongBench-Doc, while the main reported result of 53.26 accuracy / 51.19 F1 corresponds to one of the swept configurations, (k,T,a)=(5,10,5)(k,T,a)=(5,10,5). I did not find an independent development set or a pre-specified configuration-selection rule. If the final configuration was chosen after observing test performance, the reported score may be optimistic, especially because the paper itself shows that performance is highly sensitive to these budgets.The key component ablation table does not provide an exact same-budget Full controller + Full graph baseline. Table 3 fixes k=3, T=10, a=5 for its variants, but no full-system row under that exact setting is shown. This makes some component attributions difficult to interpret. In particular, the text compares the No-Search cross-page result against 34.70 from the main configuration, even though the main configuration uses a different initial-page budget. · Table 3 states that ablations use k=3,T=10,a=5k=3,T=10,a=5, but it does not include a Full-controller + Full-graph baseline under exactly the same setting. Some conclusions are instead made by comparing an ablation with the main configuration, which uses different budgets. For example, the No-Search cross-page score of 22.15 is compared against 34.70 from the main (5,10,5)(5,10,5) configuration. This confounds the contribution of search with changes in the retrieval/control budget and makes the causal interpretation of several ablations less convincing.The cost evaluation is incomplete relative to the system complexity. MAGE-RAG requires document parsing, generated abstracts, ColPali representations, graph construction, repeated controller calls, and potentially many images at the final reader. The paper reports logical input quantities for MAGE-RAG itself but no matched end-to-end latency, GPU usage, token/image cost, or preprocessing cost against the strongest baselines. · The proposed “budgeted” formulation does not tightly correspond to actual inference cost, and the efficiency evaluation is incomplete. The main cumulative budget counts activation selections, while Open, Search, and Prune operations do not consume this counter. The appendix even reports individual trajectories with around 100 opened nodes, 10 searches, and more than 100 reader images. In addition, the pipeline requires offline parsing, model-generated abstracts, ColPali embeddings, semantic-graph construction, repeated controller calls, and potentially large multimodal reader inputs. The paper reports logical input statistics only for MAGE-RAG, but no matched end-to-end latency, GPU usage, token/image cost, preprocessing cost, or serving-cost comparison against the strongest baselines. Therefore, the current experiments do not establish that the proposed budgeting mechanism provides a practical efficiency advantage.One cross-system row, MLDocRAG, is copied from its original paper rather than run in the shared reader setting. The authors disclose this, but the table should be interpreted carefully when making ranking claims. · Most reproduced baselines use Qwen3-VL-8B-Instruct, while the MLDocRAG result is taken directly from the original paper and therefore does not follow the shared-reader protocol. More importantly, the trace analysis appears internally inconsistent: samples are described as being divided into answer-level Correct and Incorrect groups, yet the logical-cost table reports 3.87 accuracy for the Incorrect group rather than zero. In addition, the budget table contains an unusual result where k=10,T=5,a=3k=10,T=5,a=3 gives 50.10 accuracy but 58.61 F1, which should be explicitly verified and explained. These issues do not invalidate the entire method, but they reduce confidence in the experimental conclusions.

Questions for authors

· How were the main budget hyperparameters selected? Was (k,T,a)=(5,10,5)(k,T,a)=(5,10,5) fixed before evaluating on MMLongBench-Doc, or was it selected after observing benchmark results? If there was no independent development set, can the authors provide a held-out evaluation or a configuration-selection procedure that does not use test-set performance?Please report the exact Full controller + Full graph result for k=3, T=10, a=5, matching the configuration used in Table 3. This is needed to cleanly interpret the component ablations. · Can the authors provide a strictly controlled same-budget ablation comparison? In particular, please report the Full controller + Full graph result under exactly k=3,T=10,a=5k=3,T=10,a=5, the setting used in Table 3. Also, why is the No-Search cross-page score of 22.15 compared with 34.70 from the different (5,10,5)(5,10,5) main configuration? Reporting all component ablations against the same full-system anchor would make the effects of search, pruning, graph structure, and node rendering much easier to interpret.Why do Open, Search, and Prune operations not consume the main action budget? Can the authors additionally report a budget measured in actual controller calls, input tokens/images, or reader-context size? · How does the proposed budget relate to real computational cost? Why do Open, Search, and Prune operations not consume the primary action budget, given that some trajectories contain around 100 opened nodes, 10 searches, and more than 100 reader images? Please also report matched end-to-end latency, total LVLM calls, token usage, image count, GPU usage, and offline preprocessing cost for MAGE-RAG and the strongest baselines under the same serving conditions.Table 5 states that examples are split into answer-level Correct and Incorrect groups, but Table 6 reports 3.87 accuracy for the Incorrect group. How exactly are these groups defined? Can the authors clarify the remaining experimental inconsistencies and comparison protocol? Table 5 describes answer-level Correct and Incorrect groups, but Table 6 reports 3.87 accuracy for the Incorrect group; how exactly are these groups defined? Please also verify the k=10,T=5,a=3k=10,T=5,a=3 result with 50.10 accuracy and 58.61 F1 and explain the large discrepancy. Finally, since MLDocRAG is taken from its original paper rather than rerun under the shared Qwen3-VL-8B setting, please clarify which cross-system ranking claims are intended to be directly comparable.

Reviewer_DSRZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAGE-RAG, an evidence-construction method for multimodal long-document-QA. It builds an offline graph containing page nodes and local multimodal elements, retrieves initial pages, and uses an online controller to activate, open, search, and prune evidence under bounded budgets. A structured renderer then assembles page context, local evidence, and images for a multimodal reader. The framework is evaluated on LongDocURL and MMLongBench-Doc, reporting 52.75 accuracy on LongDocURL and 53.26 accuracy / 51.19 F1 on MMLongBench-Doc.

Critical strengths

(1) The motivation of this paper is well grounded. The paper clearly identifies the coverage–noise–cost trade-off in fixed top-k document RAG and explains why evidence can vary in both scope and granularity. Treating evidence selection as a budgeted construction problem is a reasonable and useful formulation for multimodal long-document-QA.

(2) The method design is coherent and technically plausible. A shared page–element graph, query-specific state, explicit activation/open/search/prune actions, and page-organized multimodal rendering provide a reasonably interpretable mechanism for combining local details with page-level layout context. The trajectory and ablation analyses also make the system easier to inspect than a purely opaque retrieval pipeline.

Critical weaknesses

(1) Incomplete and insufficiently controlled empirical comparisons.The evaluation omits recent relevant systems, including URAG, DocR1, DocSeeker, and the DocV* family. Moreover, despite using a shared reader for reproduced baselines, the paper lacks systematic comparisons of model configurations, preprocessing pipelines, prompting, and inference costs. These limitations make it difficult to attribute the reported gains specifically to multigranular graph evidence construction rather than differences in models or evaluation settings.

(2) MAGE-RAG introduces multiple controller rounds and model calls, yet the paper reports only logical resource counts without controlled end-to-end latency or compute costs. It remains unclear whether the performance gains justify the additional inference, input, and preprocessing overhead.

Questions for authors
  1. Could the authors include comparisons with recent relevant methods, such as URAG, DocR1, DocSeeker, and the DocV* family, and provide a detailed configuration table for all baselines?

  2. Could the authors report end-to-end latency and computational costs, including preprocessing, model calls, and token/image usage, and compare them with direct MLLM, standard RAG, and agentic baselines under the same conditions? How much performance improvement does MAGE-RAG achieve relative to its additional cost?

  3. Could the authors clarify how sensitive MAGE-RAG is to the quality of document parsing and graph construction? To what extent do parsing errors or missing graph connections affect its performance?

Reviewer_RNpKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAGE-RAG asks how to assemble sufficient evidence for long-document QA when the relevant pages, local regions, and modalities vary with the question and reader context is limited. It builds a page–element graph, then uses a controller to activate, inspect, search for, and remove evidence before answering.

Critical strengths

None

Critical weaknesses
  1. My main concern is that I still cannot tell how much the full controller adds over the simpler graph-expansion approach. Table 3 fixes (k,T,a)=(3,10,5), but does not include Full controller + Full graph at that setting. The main result, 53.26, uses (5,10,5). This matters because Graph Neighbor Expansion reaches 52.88 and No Prune reaches 52.99 in Table 3. Those numbers do not establish that the full controller helps or fails to help.
Questions for authors
  1. What is the Full controller + Full graph result at exactly (3,10,5)? Under the same evaluation conditions, how does it compare with Graph Neighbor Expansion, No Search, and No Prune, and how stable are those differences?
  2. For the full controller and Graph Neighbor Expansion, can you report the reader-input size and model calls/tokens from the existing runs? If these differ substantially, does the controller still help under a comparable input or computation budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.