- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the distinction between retrospective oracle gains and practically learnable routing policies in adaptive RAG. It evaluates query-dependent selection of retrieval depth, generator size, and inference precision across four multi-hop QA datasets. The experiments demonstrate a substantial gap between oracle performance and learned routing, while increasing training supervision produces different effects across the three configuration dimensions.
Critical strengths
- Important evaluation distinction. Separating oracle opportunity from achievable routing performance addresses a meaningful issue in adaptive inference research.
- Controlled supervision-scaling analysis. The experiments demonstrate that additional training data improves some routing decisions but not others, highlighting differences in predictability across computational dimensions.
Critical weaknesses
- Limited methodological novelty. The primary contribution is an empirical characterization of the oracle–router gap rather than a new routing method or general theoretical framework. The conceptual distinction is intuitive, and the paper would benefit from stronger methodological or explanatory contributions.
- Limited evidence for general routing learnability. The study relies on a restricted four-configuration setting, a relatively small frozen test set of 287 queries, and estimated computational costs. The observed gap may partly reflect the selected router architecture and available features rather than an intrinsic limitation of adaptive RAG.
Questions for authors
- Does the oracle–router gap persist when using stronger routing architectures or retrieval-aware features, particularly features derived from inexpensive retrieval signals?
- How sensitive are the reported quality–cost conclusions to actual measured inference latency and a larger configuration search space?