- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a multi-agent orchestration framework for Retrieval-Augmented Generation (RAG). The authors claim that static RAG workflows and classifier-based adaptive methods, which select among a small set of predefined pipelines, cannot balance answer quality and cost across queries of varying complexity, and thus propose a planner that constructs a query-specific workflow from a set of executor modules. The Planner is trained with PPO using a reward of answer F1 minus cost and format penalties. The experiments are conducted on seven open-domain QA datasets with baselines including eight baselines.
Critical strengths
- The problem of balancing the answer quality and the inference cost in RAG systems is well-motivated.
Critical weaknesses
- The paper does not justify the benefits of multi-agent orchestration beyond end-to-end results. Moreover, empirically, without training the proposed method performs similarly with Vanilla RAG.
- The baseline comparison is not rigorous, where several baselines include a trained component, but is replaced with GPT-4o-mini. Since GPT-4o-mini cannot be trained, their reported performance likely understates the original methods.
- The novelty is limited, by stacking a trained planner with multiple standard RAG modules with fixed prompts.
Questions for authors
- Can the authors provide justification for the multi-agent orchestration beyond end-to-end experiments, e.g., a comparison against the best single fixed workflow or a simple router over the same executors?
- How were the baselines that require training implemented with GPT-4o-mini, and were they trained on the same data as MAO-ARAG?