RAG-Scholar: Context-Efficient Retrieval over Large Scholarly Corpora

Large scholarly corpora contain many passages that match a query while repeating the same concepts. Under a limited context budget, these matches can displace the source-specific details needed for an answer. We present RAG-Scholar, a hierarchical retrieval framework that separates source identification from answer evidence selection. Its graph-guided scholarly index combines explicit lexical, entity, and section-role features with a relation-aware graph neural network. Explicit features preserve query-specific source distinctions, while graph representations incorporate related descriptions from the scholarly neighborhood. Text-anchored encoding and constraint-preserving rank fusion integrate these signals across papers, sections, and local evidence. Scholarly-context-conditioned visual retrieval further discounts image regions that overlap with retained text. On a dataset constructed from 146 papers, RAG-Scholar improves Fact F1 by 16.11 points over the strongest evaluated external text baseline with a 44.6-fold reduction (97.76%) in the combined preprocessing-and-answer token count. On multimodal questions, it improves Fact F1 by 3.08 points with 71.97% fewer answer-model input tokens. Across collections of 146–597 papers, hierarchical retrieval consistently outperforms flat retrieval in paper identification and factual answering. These results support selecting compact, source-specific evidence rather than accumulating independently relevant matches. The project page and source code are available at https://rag-scholar.pages.dev.