acceptodds
Under review as a conference paper at ICLR 2027

RAG-Scholar: Context-Efficient Retrieval over Large Scholarly Corpora

Abstract

Large scholarly corpora contain many passages that match a query while repeating the same concepts. Under a limited context budget, these matches can displace the source-specific details needed for an answer. We present RAG-Scholar, a hierarchical retrieval framework that separates source identification from answer evidence selection. Its graph-guided scholarly index combines explicit lexical, entity, and section-role features with a relation-aware graph neural network. Explicit features preserve query-specific source distinctions, while graph representations incorporate related descriptions from the scholarly neighborhood. Text-anchored encoding and constraint-preserving rank fusion integrate these signals across papers, sections, and local evidence. Scholarly-context-conditioned visual retrieval further discounts image regions that overlap with retained text. On a dataset constructed from 146 papers, RAG-Scholar improves Fact F1 by 16.11 points over the strongest evaluated external text baseline with a 44.6-fold reduction (97.76%) in the combined preprocessing-and-answer token count. On multimodal questions, it improves Fact F1 by 3.08 points with 71.97% fewer answer-model input tokens. Across collections of 146–597 papers, hierarchical retrieval consistently outperforms flat retrieval in paper identification and factual answering. These results support selecting compact, source-specific evidence rather than accumulating independently relevant matches. The project page and source code are available at https://rag-scholar.pages.dev.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Hierarchical Retrieval, Graph-Based Retrieval, Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C4txOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RAG-Scholar, separating paper-level source identification from local evidence selection. It constructs concise answer contexts using hierarchical retrieval, GNN-fused academic entity representations, and text-guided visual cropping. The research problem is well-motivated: redundant topically relevant content in scholarly literature often occupies limited context space and displaces valuable evidence. The authors demonstrate promising quality–context length tradeoffs and supply thorough implementation and cost details.

Critical strengths
  1. Separating paper retrieval and evidence selection is a reasonable design. For queries constrained by affiliations, methods, or datasets, fragment-level similarity alone cannot meet source constraints.

  2. RAG-Scholar yields a 16.11-point Fact F1 gain over CompAct in Table 1, and feature fusion brings a 9.27-point Fact F1 improvement over the explicit feature branch in Table 3. These findings indicate the full system is worthy of research.

Critical weaknesses
  1. The dataset is small: 146 core papers (max 597 in extensions), and fact evaluation uses only 24 text questions and 15 visual questions. The limited question distribution and narrow focus on RAG/graph-related papers support prototype testing but fail to validate generalization to large scholarly corpora or general academic evidence synthesis.

  2. The paper’s core motivation states that independent relevance ranking preserves redundant content and displaces complementary evidence. Nevertheless, its text-centric designs, such as hierarchical filtering, source limits, graph-augmented ranking fusion, and maximum per-paper support scoring, only reduce source concentration. They do not directly measure whether a new fragment brings incremental answer-relevant information over previously selected evidence.

Questions for authors

See Weakness

Reviewer_6ferOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

When retrieving evidence to answer a question, semantic similarity-based retrieval mechanisms can retrieve relevant sources, but they may not help answer the question and could crowd out useful evidence considering the budget constraint. This paper proposes RAG-Scholar, a budget-constrained method that combines semantic similarity-based signal and GNN-learned signal to better distinguish the relevant and complementary evidence for better problem solving. When using visual content to answer the question, the authors also applied a fine-grained region selection method to focus on more relevant regions. The authors evaluated the method on their self-curated dataset which consists of 146 cs/nlp-related papers, with human-written questions and answers, and it seems that their method outperform some baselines.

Critical strengths

The paper addresses a practical problem in scholarly QA. Its hierarchical design separates paper identification from local evidence selection and limits how much context each source contributes to satisfy the budget constraints.

Critical weaknesses
  1. The method’s ability to improve evidence complementarity is not sufficiently validated. It primarily relies on semantic relevance rankings, and the GNN is trained through feature reconstruction. These components do not directly establish that the selected passages contribute distinct answer-relevant information.

  2. External validation is limited. The evaluation relies on a self-curated CS/NLP corpus.

Questions for authors
  1. Graph construction and entity matching. For the entity types in Table 6, how do you create the alias tables and matching rules? How do you match entity names in papers and queries to graph nodes, and how do you decide which nodes to connect?

  2. External evaluation on independent benchmarks, such as QASPER for paper-grounded evidence selection and ScholarQABench for cross-paper synthesis, would strengthen the empirical claims. Could you evaluate the method on an external benchmark?

  3. Why does the method improve evidence complementarity? Could you clarify how the GNN’s feature-reconstruction objective are expected to improve evidence complementarity?

  4. Generalizability. How well does the method generalize across fields and paper types using the same entity types and matching rules? For example, how does it perform on papers focused on empirical findings that may not have model entities?

Reviewer_54mXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAG-Scholar, a framework for retrieving relevant evidence from large collections of academic papers with reduced context costs. It uses hierarchical retrieval to identify relevant papers and select supporting evidence, combines explicit scholarly features with GNN representations to improve retrieval, and applies text-guided image cropping to reduce visual input. Experiments on text and multimodal question-answering tasks show promising results, achieving 69.42% Fact F1 with 97.76% fewer total tokens than CompAct on the text benchmark.

Critical strengths
  1. Well-motivated problem formulation. The paper addresses a practically important problem in scholarly retrieval: selecting relevant and complementary evidence from large collections of academic papers under limited context budgets. This is a meaningful challenge for retrieval-augmented generation systems.
  2. Coherent problem–solution alignment. The paper presents a clear motivation and a logically consistent methodology. Its proposed components directly address the identified limitations of existing approaches, particularly redundant evidence retrieval, insufficient source diversity, and inefficient context utilization.
  3. Unified retrieval framework. The paper integrates hierarchical retrieval, graph-guided scholarly indexing, and text-conditioned visual evidence selection into a unified framework. These components complement each other by improving source identification, evidence selection, and multimodal context efficiency.
Critical weaknesses
  1. Limited evaluation scale and dataset representativeness. The evaluation is conducted on a relatively small scholarly corpus of 146 papers, expanded to at most 597 papers. More importantly, Fact F1 is evaluated on only 24 of the 116 text queries and 15 of the 30 multimodal queries, raising concerns about the reliability and generalizability of the reported improvements. Although the paper describes the query categories and annotation process, it provides limited information about the actual query content and corpus distributions, making it difficult to assess the benchmark’s difficulty, diversity, and representativeness.
  2. Incomplete evaluation of the proposed visual selection mechanism. The visual ablation only compares cropped and uncropped inputs, demonstrating substantial token savings without isolating the contributions of individual components. In particular, the effectiveness of query relevance, text-conditioned redundancy penalties, and area constraints is not independently evaluated. Consequently, the benefits of the proposed redundancy-aware visual selection strategy remain insufficiently validated.
  3. Potential confounding factors in the entity-node ablations. Removing an entity type simultaneously eliminates explicit entity-matching signals and associated graph relations, making it difficult to distinguish their respective contributions. Moreover, several ablations produce unusually large drops in Fact F1 alongside substantial increases in context length. These results warrant further explanation and more controlled experiments to clarify the contributions of different retrieval components.
Questions for authors
  1. Evaluation scale and representativeness. Could the authors provide more details or representative examples of the evaluation queries to clarify their difficulty and diversity? What motivates the selection of only 24 text queries and 15 multimodal queries for Fact F1 evaluation? Additionally, could the authors provide more detailed statistics on the scholarly corpus (e.g., research topics, publication years, and document lengths) and discuss how the limited evaluation scale affects the generalizability of the results?
  2. Visual selection mechanism. Could the authors provide component-wise ablation studies to isolate the contributions of query relevance, text-conditioned redundancy penalties, and area constraints? In particular, how much of the observed benefit comes from redundancy-aware selection rather than visual cropping alone?
  3. Entity-node ablations. Could the authors further explain the substantial performance drops and increases in context length observed when certain entity types are removed? Would it be possible to separately ablate explicit entity matching and graph-based relational signals to better understand their individual contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.