acceptodds
Withdrawn from ICLR 2027

AdaMM-RAG: Adaptive Multi-Source Multi-Modal RAG via Planner-Guided Coordination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RAG grounds LLMs in external knowledge, but existing multimodal multi-agent systems usually use a fixed topology that calls all sources in parallel and aligns only vision-language evidence. This causes two issues: irrelevant sources add noise and cost, and cross-source evidence is never unified pre-fusion. Adaptive agent methods remain text-only, and do not route across sources or modalities. We propose AdaMM-RAG, which enables source-level adaptive routing via a Planner Agent that selects per-query which sources to consult. A Cross Alignment Agent unifies selected heterogeneous evidence; a Decision Agent refines and verifies answers with CoT; a quality gate triggers fallback only when evidence is insufficient. On ScienceQA, AdaMM-RAG achieves 95.51% accuracy, surpassing HM-RAG (SOTA multi-agent RAG) by +1.78% and MAO-ARAG by +12.02%; on CrisisMMD, it reaches 80.20% accuracy, outperforming HM-RAG by +21.65% and MAO-ARAG by +5.77%, while reducing retrieval cost by 55.5%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal RAG, Multi-Agent Systems, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CbtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaMM-RAG, an adaptive multi-source multimodal RAG framework that uses a planner agent to dynamically select retrieval sources based on each query. The framework further incorporates a cross alignment agent to unify heterogeneous evidence, a Decision Agent for answer refinement and verification, and a quality gate for fallback retrieval when evidence is insufficient. Experiments on two benchmarks show improvements in accuracy over existing multi-agent RAG baselines, while reducing retrieval overhead through selective source invocation.

Critical strengths

The proposed method achieves strong results on two benchmarks while selectively invoking retrieval sources, demonstrating a favorable trade-off between accuracy and retrieval efficiency. It reduces unnecessary source invocations, achieving a 55.5% reduction in normalized retrieval cost and lower end-to-end latency.

Critical weaknesses

Thanks for the authors' submission. The paper is well written and illustrates the method clearly, however, I still have several concerns about this paper:

  1. The main concern is about novelty. While the proposed framework demonstrates practical benefits, its core components largely build upon existing techniques. The planner performs prompt-based source selection, the cross alignment agent uses existing vision-language models and text-based evidence normalization, and the Decision Agent relies on LLM-based reasoning and verification. Although their integration improves efficiency and performance, the paper does not clearly establish what new methodological insights emerge from this combination. Therefore, the proposed method is more likely to be an engineering trick.
  2. Section 3.6 restricts fallback retrieval to single-source queries with insufficient evidence. However, evidence insufficiency can also occur when multiple sources are selected, particularly if the Planner omits a necessary source. The assumption that multi-source comparison provides sufficient validation is not clearly justified.
  3. The reported 55.5% cost reduction is based on normalized retrieval operation counts, which do not account for the additional LLM inference overhead introduced by the multi-agent framework. Although end-to-end latency is evaluated, the authors should further report token consumption and inference costs to provide a more comprehensive assessment of the overall efficiency gains.
Questions for authors
  1. The improvement over HM-RAG is particularly substantial on CrisisMMD, while the component ablation is only conducted on ScienceQA. Could the authors clarify which components contribute most to these improvements and whether the compared methods use consistent backbone models and inference budgets?
  2. Figure 2 shows that the Planner predominantly selects Entity and Relation retrievers, while Graph and Web retrieval are rarely invoked. How do the authors evaluate whether the selected sources are appropriate and sufficient for each query, beyond retrieval frequency and cost reduction?
  3. How does the proposed Planner compare with a straightforward LLM-based tool-selection strategy? Could a simpler router achieve comparable accuracy and efficiency improvements?
  4. Could the authors further clarify the methodological novelty of the proposed method beyond integrating existing RAG components? The planner appears to perform source selection through a prompt-based LLM call, while the alignment, decision, and fallback mechanisms primarily combine existing techniques for evidence normalization, LLM-based reasoning, and conditional retrieval. What additional benefits does the proposed multi-agent architecture provide over a simpler RAG workflow with LLM-based source routing and evidence aggregation?
Reviewer_uEQGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaMM-RAG, a multi-agent multimodal RAG pipeline built on top of HM-RAG. A training-free LLM "Planner" decides, for each query, which of five retrievers (vector, entity, relation, graph, web) to call. A "Cross Alignment Agent" turns the retrieved evidence, including image captions, into text and checks whether the sources agree. A "Decision Agent" then refines or verifies the answer with chain-of-thought. A quality gate re-plans with extra sources if a single-source answer is judged insufficient.

The authors evaluate on a 424-question subset of ScienceQA and on CrisisMMD. They report 95.51% on ScienceQA (HM-RAG: 93.73%) and 80.20% on CrisisMMD (MAO-ARAG: 74.43%). They also report a 55.5% reduction in "retrieval cost", measured as the number of retrieval calls, compared with always calling vector, graph and web. Validation consists of the main comparison tables, an ablation on ScienceQA, a three-run stability check, and a routing/cost analysis.

Critical strengths
  • The motivation is reasonable and practical. Not calling every source for every query is a sensible direction, because cost, latency and noise from irrelevant sources are real problems in multi-source RAG. The paper gives a concrete cost analysis, including a sensitivity study on the cost weights (Appendix G) and end-to-end latency (Table 3). This is the only part that clearly separates the work from HM-RAG, so it matters most for my assessment.

  • The reporting is partly honest. The authors show three-run results, admit that removing the Planner changes accuracy by only +0.23 (within the run-to-run SD), and say the low fallback rate may reflect a lenient sufficiency check. They also include the full prompts and a code link. This made it easy to check the claims, but it also exposes the weaknesses below.

Critical weaknesses
  1. Limited technical novelty, and the "on-demand" idea is not actually explored.
  • The pipeline is plan -> retrieve -> align -> decide -> verify/retry, which is a very common plan-then-act template. Every component is an LLM prompt: the Planner scores five sources from 0 to 3, and the alignment and decision agents are also prompts. Compared with HM-RAG, the new parts are essentially a routing prompt and an extra stage.
  • On-demand retrieval is the most interesting idea, but here it is reduced to a one-shot choice of sources before any retrieval happens. The paper does not study how to identify what a query really needs, or when to retrieve (e.g., after seeing partial evidence or in the middle of reasoning).
  • There is no evaluation of routing quality: no oracle or gold source labels, no comparison with simple heuristics or learned routers at equal cost, and no comparison with Self-RAG/Adaptive-RAG-style decisions.
  • The Planner must pick at least two sources except for trivial lookups, and Appendix B shows that 58-89% of queries use exactly two sources. This looks more like "dropping a few sources" than adaptive routing.
  • The ablation supports this reading. The Planner gives +0.23 accuracy (within noise), and the quality gate fires on only 4.2% of queries. The early exit and the gate themselves are never ablated. So the claimed contributions on routing and the quality gate are weakly supported.
  1. The evaluation is too narrow and not clean enough to support the SOTA claims.
  • Only two benchmarks are used. ScienceQA is nearly saturated, and most questions can probably be answered without external retrieval (the web source is called only 93 times). CrisisMMD is tweet classification. There is no knowledge-intensive multimodal benchmark where source choice really matters.
  • There is no no-retrieval baseline with the same backbone (Qwen3-8B / Qwen3-VL-4B, direct answer and CoT). The gains may come from a much newer backbone than the GPT-3/ChatGPT-era baselines, not from retrieval coordination.
  • Numbers for the proposed method are the best of three runs (Sec. 4.2), while the mean is 95.20 and 80.00. The baselines appear to be single runs.
  • ScienceQA results are on a 424-question subset, but Table 1 compares them with leaderboard numbers that, as I understand it, come from the full test set. The subset sizes are also inconsistent: NAT+SOC+LAN = 424 and G1-6+G7-12 = 424, but TXT+IMG+NO = 644. The "1,492 queries" in the cost analysis equals the sum over all eight overlapping category splits, so it is not the number of unique queries.
  • The method is clearly worse than HM-RAG on LAN (90.65 vs. 94.91) and worse than fine-tuned models on NAT. This is not discussed.
  1. Presentation: incomplete table, unclear figure, and text that does not match the implementation.
  • Table 1 has many missing entries. ChatGPT, CoT GPT-3+Doc, LaVIN-13B and both LG-VQA rows (BLIP-2, CLIP) report only the average, with all subset metrics missing, so a per-category comparison is impossible.
  • Figure 1 is a raster image with very small text. Please replace it with a vector/PDF figure.
  • Several parts read like unedited AI-polished text and do not match what is actually done:
    • Sec. 3.4 and Eq. 5 describe alignment as projecting all evidence into a shared embedding space, with a "lightweight string-matching or embedding-similarity" consensus check. Appendix F.2 and Figs. 3-4 show it is an LLM prompt that filters evidence, resolves conflicts and writes a summary context.
    • Eq. 4 talks about a projection layer, Q-Former and frozen decoder, but the implementation seems to use an off-the-shelf Qwen3-VL-4B captioner.
    • The index set in Eq. 5 ({V, I, G, W}) differs from the five sources used elsewhere.
    • "Contradict/Insufficient are as defined above" (Sec. 3.5) is never defined.
    • Appendix F refers to an "interpretability analysis in Section 4.4" that does not exist, and Appendix G refers to per-category call counts "in Appendix C" that are not there.
  • Using AI to polish the writing is fine, but the authors should check that every claim and equation matches what the code does.
Questions for authors

see weakness

Reviewer_2hC6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaMM-RAG, a multimodal, multi-agent RAG framework that adaptively selects retrieval tools for each query. A training-free Planner Agent chooses a subset of vector, entity, relation, graph, and web retrievers. The retrieved evidence is normalized into a shared text space, where a Cross Alignment Agent filters irrelevant information and resolves conflicts. A Decision Agent then refines the answer rather than relying on majority voting, while a quality gate triggers additional retrieval when single-source evidence is judged insufficient. Experiments on ScienceQA and CrisisMMD report improvements over several single-agent and multi-agent baselines, together with reductions in retrieval calls and latency.

Critical strengths
  1. The paper addresses a practical and important problem. In realistic RAG systems, indiscriminately invoking every available retriever can incur unnecessary computational cost and expose the generator to irrelevant or conflicting evidence. Query-dependent coordination across heterogeneous retrieval tools is therefore a meaningful direction to explore.
  2. The proposed framework is modular, intuitive, and potentially practical because the Planner does not require additional training. The evaluation considers two multimodal benchmarks and reports ablations, multi-run stability, latency, and retrieval cost. In particular, the results provide evidence that the Planner can reduce latency and that the Decision Agent contributes meaningfully to final accuracy.
Critical weaknesses
  1. The proposed alignment mechanism introduces a strong but insufficiently justified text bottleneck. Images are converted into captions, graph triples are verbalized, and web and textual evidence are normalized before being processed through a shared text representation. Although this facilitates comparison across sources, it may discard fine-grained visual information and graph structure. The current ablation treats captioning, graph verbalization, relevance filtering, conflict resolution, and context synthesis as a single Alignment component. Moreover, the Alignment component is only removed when the Decision Agent is also absent. Consequently, the experiments do not establish whether converting all heterogeneous evidence into text is preferable to native multimodal or structure-preserving fusion.
  2. The novelty and causal empirical support for the proposed components are not sufficiently established. The paper does not clearly explain which individual technical element is novel beyond the specific integration of a Planner, multiple retrieval tools, text-based alignment, a Decision Agent, and a quality gate. The distinction from executor-level routing is also unclear because vector, entity, relation, graph, and web mix retrieval mechanisms, graph operators, and actual data sources. Experimentally, removing the Planner changes accuracy by only 0.23 points, which the paper acknowledges is within the reported run-to-run variation, although it substantially affects latency. There is also no direct comparison against majority voting, no dedicated validation of the consensus-based early exit, and no ablation or recovery analysis for the quality gate. Thus, the strong end-to-end results do not clearly identify which claimed components are responsible for the improvements.
Questions for authors
  1. Can the authors provide a direct comparison between the proposed shared-text alignment and alternatives that preserve native modality or structure, such as allowing the Decision Agent to access the original image, retaining graph-native representations, or simply concatenating source-specific evidence? Can the individual effects of captioning, graph verbalization, relevance filtering, conflict resolution, and context synthesis be separated?
  2. What is the precise technical novelty of the proposed framework beyond integrating several RAG components into a single pipeline? Are entity, relation, and graph genuinely distinct data sources, or are they different retrieval operators over the same underlying knowledge graph? Please also clarify the underlying corpora and indices used by each of the five retrievers.
  3. Can the authors provide mechanism-level evaluations of the Planner, consensus-based early exit, and quality gate? In particular, it would be useful to report source-selection quality relative to an oracle, the precise definition and threshold of Agree, the accuracy and wrong-consensus rate of early-exit cases, and how often fallback retrieval corrects an initially wrong or insufficient answer. A direct comparison between the Decision Agent and majority voting would also help validate the claimed advantage of refinement over voting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.