acceptodds
Under review as a conference paper at ICLR 2027

Agents Don't Need a New Index: A Scientific Knowledge Layer over Lexical Search

Abstract

The web is increasingly accessed by AI agents rather than humans. Every agent needs knowledge, and nowhere more so than in science, where agentic pipelines are proliferating. Large scientific corpora are already reachable through mature lexical-based search engines, but these interfaces are not built for AI agents: they expect keywords and complex syntax and return whole papers, so every agent must learn the query syntax, issue several searches, and read entire articles to find the evidence it needs. We introduce LIBRARIAN, a knowledge layer that upgrades lexical-based search interfaces for AI agents: an agent asks in natural language and receives evidence that answers it. A single LLM orchestrates the whole knowledge retrieval process: it plans complementary subqueries executed by the search engine, then reads the selected papers and locates the relevant evidence. We instantiate LIBRARIAN on Europe PMC, an open lexical-based search engine that provides comprehensive access to scientific literature, and we evaluate multiple AI agents with access to LIBRARIAN across four scientific benchmarks: literature synthesis, claim verification, open-form question answering, and downstream biology tasks such as protocol questions and sequence manipulation. On ScholarQABench, agents equipped with LIBRARIAN improve Citation F1 by more than points over strong recently published baselines. Used as the retrieval layer of an existing claim-verification pipeline, it increases agreement with expert consensus by points, and on the open-form LitQA2 benchmark, a GPT-5.4 agent scores about points higher when using the LIBRARIAN rather than web search. Our results show that existing knowledge infrastructure can be reused in AI pipelines without reindexing corpora or sacrificing performance. We release LIBRARIAN openly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, scientific research, scientific agents, retrieval-augmented generation, information retrieval, agentic search, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hdZpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LIBRARIAN, which expands a natural language question into complementary literature queries, locates passages in retrieved papers, and uses a language model to select and rank citable sentences. The authors connect it to agents for literature synthesis, scientific claim verification, open form question answering, and biology tasks, then compare their performance.

Critical strengths

The “Semantic evidence selection” ablation compares semantic selection with lexical ranking over the same candidate passage pool and evidence length limit, and finds clear differences in both open form question answering and literature synthesis. This control targets a central part of the method and gives more direct evidence for the value of fine grained selection than end to end scores alone.

Critical weaknesses

The claim verification results table compares the original ProClaim system with ProClaim using LIBRARIAN, and the main text interprets the higher agreement as better access to knowledge through retrieval. Yet the appendix's ProClaim implementation details explain that adaptation also changes fact extraction from reading full papers to labeling selected evidence directly, and replaces the original sufficiency classifier with a language model alternative. The original result is also taken from prior work. These changes could themselves affect which facts are selected, whether retrieval continues, and the final verdict. The comparison therefore does not isolate the retrieval layer's contribution or establish the claimed causal interpretation for this task.

The literature synthesis results table mainly reports citation support for the biomedical and neuroscience tasks. That measure checks whether cited sources support the statements the agent wrote; it cannot tell whether important content was omitted. In the “Full-text access” appendix ablation, restricting evidence to abstracts changes this measure only slightly despite the substantial loss of full text, and the authors acknowledge that it reflects consistency between claims and citations more than answer quality. A separate coverage related score on the multidisciplinary subset does not directly validate the gains on the main tasks or their fixed corpus comparison. Thus, the central interpretation that literature synthesis quality improves still lacks a measure aligned with completeness.

Questions for authors

Could you provide or identify a comparison that keeps ProClaim's fact extraction and sufficiency decision procedure fixed while changing only the retrieval layer, so the retrieval benefit can be separated from the downstream workflow changes?

Could you assess coverage or completeness of key facts for the main literature synthesis tasks under the same corpus and generation settings, to show whether the gain in citation support also means more complete synthesis?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hdZp

We thank the reviewer. Each of the two observations is answered below with a controlled comparison.

W1 and Q1. Isolating the retrieval layer in claim verification

Only the retriever changesClaimsPubMed + S2LIBRARIAN
One-prompt Verifier Agent, same prompt and verdict model4060.530.68p < 0.001
ProClaim's loop; the extractor reads the evidence LIBRARIAN selects4060.580.57p = 0.67
ProClaim's loop; its own sub-agent reads the full text of the papers found1000.610.54p = 0.17

The cleanest test of the retrieval layer is the Verifier Agent, where the agent, prompt, and verdict model are identical and only the retriever changes: LIBRARIAN raises agreement with expert consensus from 0.53 to 0.68 across all 406 labeled claims (419 in the submission were with an old version of the released dataset).

We also ran the comparison you ask for inside ProClaim. The last row keeps ProClaim's fact extraction and sufficiency decision unchanged: LIBRARIAN only finds the papers, and ProClaim's own sub-agent reads their full text. The two retrievers do not differ significantly. The lower figure reflects our one-hour limit per claim: reading whole papers is slower and this arm ran on a slower GPU, so 39 claims reached the limit against 25, and they count as wrong. On the 52 claims that every arm answered in time, agreement is the same. On all 406 claims (second row) the retrievers are on par. So inside a loop designed around PubMed and Semantic Scholar, LIBRARIAN replaces both at no loss in agreement, and in a simple agent it raises agreement from 0.53 to 0.68.

As you correctly noted, rows 0.79 and 0.84 of Table 3 also differ in fact extraction and in the sufficiency classifier, and that 0.79 is taken from the Proclaim pipeline, so the comparison does not isolate the effect of the retriever. What it does show is a consequence of the evidence LIBRARIAN returns: because that evidence is already selected for relevance to the claim, it can be processed in a much simpler way than in the original agent, and this is enough to reach 0.84. The revised table labels the two rows accordingly, and the text presents the Verifier Agent as the retrieval result. In every arm above, the sufficiency decision is made by the LLM classifier that ProClaim provides, since its MLP needs impact factor, citation counts, and full-text features that LIBRARIAN's evidence does not carry.

W2 and Q2. Completeness on the main synthesis tasks

We measured it on the main tasks, with the corpus (OpenScholar's) and the generator (GLM-5.3-Flash) held fixed. LIBRARIAN's answers score higher than BM25's on coverage and organization, contain more supported statements from more sources, and are on par with the dense pipeline's; relevance is the same for all three (Table 13, Appendix C.2).

BM25Dense retriever + rerankerLIBRARIAN
Prometheus coverage (1–5)4.22 †4.424.46
Prometheus organization (1–5)4.34 †4.52 †4.69
Prometheus relevance (1–5)4.114.164.15
Supported statements per answer, Bio / Neu16.0 † / 13.7 †–24.5 / 22.1
Share of citation-worthy statements that are supported, Bio / Neu0.76 / 0.74–0.83 / 0.84
Distinct sources cited per answer14.3–24.5
Words per answer605665836

† significantly different from LIBRARIAN. Prometheus rows and word counts: 100 Bio and 100 Neu questions. Statement rows: full splits.

Each answer contains about 8 more supported statements, drawn from 10 more sources, and a larger share of its citation-worthy statements is supported, a ratio that does not depend on answer length. In a pairwise comparison of key-fact coverage (GPT-4o judge, both orders), LIBRARIAN's answer is preferred to BM25's on 136 of the 200 questions, tied on 44 and not preferred on 20.

LIBRARIAN's answers are also longer, and the gain in Prometheus coverage is concentrated in the questions where they are longer. We therefore present the result as more strongly supported by more sources, and Sec. 3.2 now says explicitly that Citation F1 does not measure completeness. Against the dense pipeline, coverage is the same.

To Reviewer hdZp: Both points are now covered by a controlled comparison: the retriever is isolated in claim verification, and completeness is measured on the main tasks with the corpus and generator fixed. We respectfully request a re-evaluation in light of these results.

AuthorsOpenReview ↗

Kind Reminder: Discussion Deadline Approaching

Dear Reviewer hdZp,

Thank you again for your time and the detailed, constructive feedback you provided on our paper.

As the discussion period deadline is approaching, we wanted to follow up. We have submitted a detailed rebuttal addressing the concerns you raised, including additional experiments and analyses. To summarize, in our response, we have:

  • Provided a controlled comparison on ProClaim, keeping fact extraction and the sufficiency decision fixed while varying only the retriever (Q1, Tables 3 and 14).
  • Conducted additional completeness evaluations on the main synthesis tasks, keeping the corpus and generator fixed to ensure a fair comparison (Q2, Table 13, Appendix C.2).

Both points you raised are now covered by a controlled comparison.

We respectfully ask that you reconsider your rating in light of these results, and we welcome any further questions before the imminent deadline.

Thank you for your response. It addressed my concerns, and I have decided to increase the score.

We thank the reviewer for reconsidering the paper and for raising the score. We are glad that the two controls you asked for, the retriever isolated in ProClaim and completeness on the main synthesis tasks, addressed your concerns.

Since no concerns remain open, we would be grateful if you would consider whether the paper now merits a clear acceptance. If anything still holds you back, please let us know, and we will address it before the discussion closes.

Reviewer_RqcwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LIBRARIAN turns the existing Europe PMC search infrastructure into a retrieval service for scientific agents. It uses LLM-generated queries, lexical passage ranking, and LLM-based evidence selection to return cited snippets without maintaining a separate dense index. I see real practical value in that design. The main question is whether the paper establishes a distinct technical contribution, rather than a well-engineered combination of familiar retrieval components.

Critical strengths

The system solves a concrete integration problem and returns evidence at a useful level of granularity. It also has some convincing within-setting results: with the same GLM-5 model and OSDS corpus, Citation F1 improves by 4.2 points on ScholarQABench Bio and 5.4 on Neu compared with BM25. The ablations also make clear that LLM evidence selection matters. These results support the usefulness of the pipeline, even if they do not settle the novelty question.

Critical weaknesses

1.

The methodological distinction from existing retrieval pipelines is unclear. Query expansion, BM25 ranking, and LLM evidence filtering are established ideas. Packaging them as a reusable service over Europe PMC is useful, but the paper should identify which design choice is new and demonstrate why it matters. A matched comparison against multi-query BM25 with an equally strong LLM reranker, or an agent using the Europe PMC API directly, would be more persuasive than comparisons across different models or corpora.

2.

Several highlighted gains are affected by changes beyond retrieval. The larger ScholarQABench improvement over historical systems partly reflects the use of a stronger generation model; the same-model gains above are the cleaner evidence. For ProClaim, the reported change from 0.79 to 0.84 also comes with a different evidence-processing workflow and a replacement of the original MLP sufficiency classifier with an LLM classifier (Appendix C.2). In LAB-Bench, the retrieval condition receives additional system-level instructions. Neither comparison cleanly isolates the effect of LIBRARIAN. The paper should compare retrieval conditions while keeping the downstream decisions and prompts fixed.

The claimed advantage of fielded search is not established by the ablation. On LitQA2, adding structured field queries changes accuracy from 36.6 to 39.9, with overlapping confidence intervals (Table 7). By contrast, replacing LLM evidence selection causes a much larger drop. That pattern suggests evidence selection may be the principal driver of performance, rather than exploiting structured fields in the existing index. A targeted benchmark of entity- and field-constrained questions would be a better test of this distinctive feature. The efficiency claim also needs tighter framing: the cost measurements use GLM-5.3-Flash rather than the GLM-5 configuration in the main experiments, and reusing Europe PMC shifts some indexing costs to an external service.

LIBRARIAN looks like a practical system with measurable benefits. For an ICLR paper, though, the current experiments do not clearly establish what is new or which component deserves credit for the gains. I would rate it Weak rejection (2/4).

Questions for authors
  1. What remains of the gains against a budget-matched multi-query BM25 plus LLM-reranking baseline on the same corpus and model?
  2. Could the ProClaim and LAB-Bench comparisons be rerun with the same downstream classifier, evidence processing, and prompts in both conditions?
  3. On which types of queries does fielded search reliably beat unfielded search? What is the end-to-end cost of the exact configuration used for the main results?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Rqcw (part 1 of 2)

We thank the reviewer. We ran both baselines you proposed (W1, Q1).

An agent using the Europe PMC API directly. GPT-5.4 with the Europe PMC search and full-text endpoints as tools (up to 8 calls, no LIBRARIAN component; row h) answers 44.0% of LitQA2 questions correctly. Through LIBRARIAN, the same model answers 72.5% of the same questions on the same seed, a significant gain of 28.6 points. It reads 612 tokens of evidence instead of 11,100 and costs about USD 0.01 per question (LIBRARIAN on GLM-5.3-Flash plus the GPT-5.4 answer) instead of USD 0.10 (GPT-5.4 throughout). With GLM-5.3-Flash in place of GPT-5.4 in both arms, the figures are 49.1% and 68.5%.

Multi-query BM25 with an equally strong LLM reranker (row c). Same planned queries, candidates and LLM as LIBRARIAN; it returns the top 16 paragraphs instead of selected sentences. Table 4 of the revised paper (rows b–g share one candidate pool):

MethodQueriesUnitLimitLitQA2 Acc.Tok.Bio F1Cov.Org.CitedTok.Papers
(a) BM25QuestionParagraph16 paras.27.53,28477.9––5.13,1769
(b) BM25LLMParagraph16 paras.67.04,51875.94.454.2310.44,12614
(c) RerankerLLMParagraph16 paras.77.74,63582.84.494.3510.84,45112
(d) RerankerLLMParagraph= LIBRARIAN76.260080.94.484.5115.38,53721
(e) LIBRARIANLLMSentenceNone76.661277.74.464.6120.78,87444
(f) LIBRARIANLLMSentence20k chars76.661280.84.554.4312.94,04614
(g) LIBRARIANLLMSent.→para.20k chars78.41,73685.14.464.288.04,0089
(h) No layer: GPT-5.4 agent on the Europe PMC API (1 seed)44.011,100––––––

LitQA2. LIBRARIAN matches the reranker (76.6% against 77.7%, not significant) with 8x less evidence. The budget-matched reranker (d) also ties (76.2%), but that budget is not its own: it receives, for each question, the amount of evidence LIBRARIAN chose.

ScholarQA: Citation F1 follows the number of sources. The reranker has the higher Citation F1 (82.8 against 77.7). Among the LLM-selected rows, Citation F1 broadly falls as the number of source papers grows: 85.1 (g, 9 papers), 82.8 (c, 12), 80.8 (f, 14), 80.9 (d, 21), 77.7 (e, 44). The mechanism we observe: with more sources, the generator attaches 2.7 citations per sentence instead of 1.9, and a citation counts as an error when its source alone does not support the sentence and the other cited sources do. What the answers contain does not follow F1: coverage stays between 4.45 and 4.55 in rows (b)–(g) while F1 spans 75.9 to 85.1; single-query BM25 (a), the weakest retriever (27.5% on LitQA2), scores 77.9 from 9 papers; and capping LIBRARIAN at 2,500 characters raises F1 to 84.5 while coverage and organization fall below 4 (Table 11). On content, LIBRARIAN's answers are as complete as the reranker's (coverage 4.46 against 4.49), better organized (4.61 against 4.35, significant), and cite 20.7 distinct papers against 10.8.

At the reranker's budget, the gap disappears. With about the same amount of evidence (4,008 against 4,451 tokens), returning the paragraphs that contain LIBRARIAN's selected sentences (g) reaches 85.1 Citation F1 against 82.8 for the reranker (significant), with the same coverage (4.46 against 4.49), and 78.4% on LitQA2 (77.7% for the reranker) with 2.7x less evidence. The lower Citation F1 of row (e) therefore comes from how much evidence it passes, not from what LIBRARIAN selects. Capping trades breadth for precision (organization 4.28; 8.0 papers cited); the main results use the uncapped output (e).

Which design choice is new, and why it matters. What the paper establishes is the pipeline itself: LLM query planning in the native syntax of an existing search service, followed by LLM evidence selection, takes that service from 27.5% to 76.6% on LitQA2, with no index of its own. On ScholarQABench it matches OpenScholar's dense retriever and reranker (Table 2, last block).

W2(a) and Q2. ScholarQABench with only the retriever changing

The same-model gains that you call the "cleaner evidence" hold under stricter controls. With corpus and generator fixed (OpenScholar's corpus, GLM-5.3-Flash, full splits), LIBRARIAN exceeds BM25 by 6.0 points on Bio and 8.7 on Neu, both significant, and it matches the dense retriever and reranker on Bio and exceeds it on Neu. The gain does not depend on the generator: with GPT-4o writing from the same retrieved passages, it is 8.3 and 8.2 points (100 questions per split). As you noted, the generator matters (on identical passages, GPT-4o scores 17 to 25 points below GLM-5.3-Flash), so the revised text presents the comparison with historical systems as indicative only.

Part 2 follows.

AuthorsOpenReview ↗

Reply to Reviewer Rqcw, part 2 of 2 (W2b, W2c, W3, Q3)

W2(b). ProClaim with only the retriever changing

PubMed + S2LIBRARIAN
Verifier Agent (one prompt, same verdict model)0.530.68p < 0.001
ProClaim's own loop (same LLM, extraction model and sufficiency classifier)0.580.57p = 0.67

In the Verifier Agent, only the retriever changes, and LIBRARIAN raises agreement from 0.53 to 0.68 (to 0.63 when it runs on GLM-5.3-Flash, also significant). In ProClaim's own loop, LIBRARIAN performs on par with PubMed + Semantic Scholar: one natural-language call replaces the two search services the loop was built around, at no loss in agreement. Both arms use the LLM sufficiency classifier that ProClaim provides, since its MLP relies on features (impact factor, citation counts, full text) that LIBRARIAN's evidence does not carry; both run on GLM-5.3-Flash, instead of Claude Sonnet 4.6 as reported in the paper. Both arms score below the 0.79 reported in the paper because 117 and 120 of the 406 claims reach our one-hour limit, and these count as wrong. The cause is ProClaim's own pipeline, not GLM-5.3-Flash: in many of these claims, the fact extractor, which we kept as in the original (Qwen), returns no facts, and the loop keeps searching until the limit. Appendix C.3 reports these changes, highlighted in the revised PDF.

We agree that rows 0.79 and 0.84 in Table 3 differ beyond the retriever, and the table now labels them accordingly. What 0.84 shows is a consequence of the evidence LIBRARIAN returns: because it is already selected for the claim, it can be processed more simply than in the original agent.

W2(c). LAB-Bench with the prompt matched.

We ran the experiments you described, LIBRARIAN's prompt with an empty evidence block:

Base, plain promptLIBRARIAN's prompt, no evidenceLIBRARIAN's prompt + evidence
GPT-5.4, four subsets (1,261 questions)46.555.754.2
GPT-5.4, SeqQA (600)50.864.565.8
GPT-4o, four subsets (1,261)32.829.327.3

The control confirms the confound you identified. The instruction alone raises GPT-5.4 from 46.5 to 55.7 (on SeqQA from 50.8 to 64.5, significant), and adding the evidence changes almost nothing. The gain on sequence tasks that the submission attributed to LIBRARIAN comes from the answering instruction, more than the retrieved evidence. We have corrected the claim: it is removed from the abstract, introduction, contributions, Figure 1 and the discussion, and Sec. 3.4 states the correction.

LAB-Bench is now the paper’s control (Sec. 3.4); Table 15 in Appendix C.5 provides all subsets for both models, with a paired test for each effect. These four subsets test sequence manipulation, database lookup, protocol troubleshooting, and cloning, whose answers are not in the literature. With the same model (GPT-5.4), retrieval adds 60 points on LitQA2, where the answer is in the literature (17.6% to 78.0%, Table 1), and nothing beyond the prompt on these tasks, where it is not.

W3 and Q3. Fielded search.

Fielded sub-queries are how LIBRARIAN reaches a third of the evidence it uses. In the logs of the main runs, 44% of the sub-queries on ScholarQABench use field syntax (title/abstract, keyword, gene/protein, disease, methods), and 33% of the papers cited in the answers are reached only by a fielded sub-query; on LitQA2 the share of evidence papers is 24% (Table 12). We agree that the increase from 36.6 to 39.9 on LitQA2 does not constitute an accuracy gain. LitQA2 is the benchmark least able to show one, because each question is answered by a single paper. A targeted, entity-constrained benchmark is the right test, and we could not build it within the rebuttal period, so the revised text states that the accuracy benefit of fielded queries is not yet established.

W3 and Q3. Cost of the main configuration.

We measured cost on the run that produces the GLM-5.3-Flash rows in Table 2, so cost and results refer to a single configuration: USD 0.0088 per LIBRARIAN call (52K tokens) and USD 0.0118 per ScholarQABench question end-to-end, compared with USD 0.0020 with BM25. At GLM-5's list price, the same tokens would cost about 6.6 times as much.

On reusing Europe PMC: this is the design. LIBRARIAN reuses an index that a public service already builds and maintains for its own users, so an agent adds query load and no indexing cost; the revised text says so, and the limitations report the latency that comes with a shared service.

To Reviewer Rqcw: The soundness concern was that the gains were not isolated from other changes. Every comparison now changes only the retriever; both baselines you proposed are in the paper, and each claim follows the controlled result. We respectfully request a re-evaluation of soundness in light of these controls.

AuthorsOpenReview ↗

Kind Reminder: Discussion Deadline Approaching

Dear Reviewer Rqcw,

Thank you again for your time and the detailed, constructive feedback you provided on our paper.

As the discussion period deadline is approaching, we wanted to follow up. We have submitted a detailed rebuttal addressing the concerns you raised, including additional experiments and analyses.

To summarize, in our response, we have:

  • Provided new comparisons against a budget-matched multi-query BM25 + LLM reranker and an agent directly calling the Europe PMC API (Q1, Table 4).
  • Rerun ProClaim and LAB-Bench with the downstream decisions and prompts fixed (Q2, Tables 3, 14 and 15). The LAB-Bench control confirmed the confound you identified, and we corrected the claim in the paper.
  • Reported how fielded queries are used in the main runs (Table 12) and stated that their accuracy benefit is not yet established (Q3).
  • Provided a detailed cost analysis of our main configuration (Q3, Appendix A.5).

Your concern about soundness was that the gains were not isolated from other changes. Both baselines you proposed are now in the paper, and the claims follow the controlled results.

We respectfully ask that you reconsider your rating in light of these controls, and we welcome any further questions before the imminent deadline.

Reviewer_cYurOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Librarian, a scientific literature retrieval layer that lets agents query an existing lexical search engine in natural language and receive citable evidence. It combines LLM-generated search queries, BM25 paragraph retrieval, and LLM-based evidence selection without maintaining a separate dense index. Experiments across four benchmark suites evaluate its usefulness for literature synthesis, claim verification, scientific question answering, and biology tasks.

Critical strengths

The system produces gains across several benchmarks and downstream agents, supporting its practical usefulness.

Critical weaknesses
  1. The methodological contribution is limited. Query expansion, BM25 paragraph retrieval, and LLM-based evidence selection are established techniques, and their integration here is straightforward. Although the paper clearly lists its contributions, it does not establish a substantial methodological advance or a sufficiently new empirical finding beyond demonstrating that this combination works well.

  2. The cost–performance tradeoff is insufficiently established. Appendix A reports an illustrative dense-index infrastructure comparison, but it does not compare retrieval quality and total cost against strong dense or hybrid systems under matched conditions.

Questions for authors
  1. How does Librarian compare with a strong dense or hybrid retrieval pipeline in quality, latency, and total cost under the same corpus and backbone? Can you report costs for the configuration used in the main experiments?

  2. Which specific design choice distinguishes Librarian from a standard query-expansion, retrieval, and reranking pipeline? The ablations establish the usefulness of individual components, but can you clarify what new methodological or empirical insight their combination establishes beyond existing approaches?

6 comments
AuthorsOpenReview ↗

Response to Reviewer cYur

We thank the reviewer. We answer both questions with new experiments.

W1 and Q2. What is new, and what distinguishes LIBRARIAN from a query-expansion, retrieval and reranking pipeline

To our knowledge, this is the first work to show that one knowledge layer returning evidence, not entire papers, can serve different kinds of scientific agents with no index of its own: it queries an existing, publicly maintained search service instead. OpenScholar, SciRAG, and PaperQA2 build, host, and refresh a dense index over a snapshot of the literature. LIBRARIAN retrieves from a search index that a public service already optimizes, keeps current, and annotates with search fields (gene/protein, disease, methods section, and others), and an LLM operates it. The new experiments show three things.

  1. It gives up no quality. With the same generator, LIBRARIAN matches or exceeds OpenScholar's dense retriever and reranker, and on live Europe PMC it does so with no index at all (table below).
  2. The layer is what makes the engine usable by an agent. On LitQA2, Europe PMC queried with the question itself gives 27.5%, GPT-5.4 calling the Europe PMC API as an agent gives 44.0%, and LIBRARIAN gives 76.6%, at about a tenth of the agent's cost.
  3. One layer serves different agents with no task-specific index or training: literature synthesis (Table 2), factoid questions (17.6% → 78.0%, Table 1) and claim verification (0.53 → 0.68, Table 3).

Two design choices distinguish it from a standard pipeline. The first is where it retrieves: the LLM writes its queries in the fielded syntax of an existing service and ranks what that service returns, so there is nothing to embed, host, or refresh. The second is what it returns: a single LLM call selects sentences and decides how much evidence the question needs, whereas a reranker returns a fixed number of paragraphs. We tested the second choice against multi-query BM25 with an equally strong LLM reranker (new Sec. 3.5, Table 4). On LitQA2, LIBRARIAN achieves the same accuracy (76.6% vs. 77.7%) using 612 evidence tokens instead of 4,635. For a synthesis question, it widens the evidence to 44 papers, where the reranker passes 12 (Citation F1, which favors short contexts, is 82.8 for the reranker and 77.7 for LIBRARIAN). Query planning contributes most on factoid questions (27.5% → 67.0%), and LLM evidence selection helps on both benchmarks.

W2 and Q1. Quality, latency, and cost against a dense or hybrid pipeline, same corpus and backbone

We ran this comparison on OpenScholar's own corpus (237M passages). The 149M in the submission was an error, and all OSDS rows use this index, with the same generator (GLM-5.3-Flash) and scorer, across the full Bio and Neu splits. The dense rows use OpenScholar's released retriever (exact search over its released embeddings), alone and with its released reranker.

RetrieverIndex it needsCitation F1, BioCitation F1, NeuUSD per question
BM25lexical index68.868.50.0020
Dense retriever, no reranker365 GB of embeddings in memory70.669.20.0020
Dense retriever + reranker365 GB of embeddings in memory74.273.00.0023
Hybrid (BM25 + dense) + rerankerlexical index + embeddings72.773.00.0023
LIBRARIAN, same corpuslexical index only74.877.20.0157
LIBRARIAN, live Europe PMCnone74.476.20.0118
  • Quality. LIBRARIAN exceeds the dense pipeline on Neu by 4.2 points, matches it on Bio, and exceeds the hybrid on both. Without the reranker, dense retrieval adds only 1.7 and 0.6 points over BM25: the reranker gives most of the gain.
  • Index. The dense pipeline holds 365 GB of embeddings in memory (728 GB at full precision); LIBRARIAN on Europe PMC holds none.
  • Cost. LIBRARIAN costs USD 0.0095 more per question, because its LLM reads 66K input tokens instead of 10K. A dense pipeline pays instead for a machine that holds the embeddings all year. We search them exactly, as OpenScholar's released code does. On a CPU host with 512 GB of memory (about USD 37k a year), where a query takes tens of seconds, owning the index pays off above about 4 million questions a year; for sub-second search on five H100 GPUs (about USD 301k), above about 32 million. Neither figure includes building or refreshing the index.
  • Latency. A dense pipeline answers in about 0.3 s on GPU (our estimate). LIBRARIAN makes two LLM calls and waits on the search service: 23 s at the median on Europe PMC (Table 6: 12 s in LLM calls, 9 s in Stage 2, mostly waiting on Europe PMC).
  • Main configuration. Costs are from the run behind the GLM-5.3-Flash rows of Table 2 (USD 0.0088 per LIBRARIAN call, 52K tokens).

To Reviewer cYur: The matched dense-retrieval comparison that both concerns asked for is now in the paper, and shows that LIBRARIAN matches the quality of a dedicated dense index. We respectfully request a re-evaluation.

Thank you for the detailed response and the new experiments. After careful consideration, I decide to keep my score.

  • The same-corpus comparison with dense and hybrid retrieval helps clarify the cost–performance trade-off. However, the result reads as comparable quality at a notably higher per-question cost and latency, traded for a lower fixed cost.
  • The new ablation (Table 4) does not address my concern about methodological novelty. A plain LLM reranker over the same planned queries matches or outperforms LIBRARIAN, and fielded queries show no measurable benefit. This reflects a well-engineered combination of known components rather than a new retrieval method.
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for the reply. We respectfully disagree with both points as stated, because each leaves out results in the revised paper.

1. "Comparable quality at a notably higher per-question cost and latency"

This summary does not match Table 7.

  • Quality is not only comparable. On the same corpus, LIBRARIAN exceeds the dense retriever + reranker on Neu by 4.2 points (95% interval [+3.4, +5.0]) and the hybrid on both splits (+2.1, +4.2). Only Bio is a tie.
  • "Notably higher per-question cost" is USD 0.0095. The dense pipeline is cheaper in total only above about 4 million questions per year on a CPU host, or 32 million on a GPU host, before building and refreshing the index (about 0.5 million with a compressed index, at an unmeasured recall loss).
  • Lower latency holds only for the GPU host (five H100s, about USD 301k per year). On the CPU host, exact search takes tens of seconds per query (our estimate), like LIBRARIAN.

W2 asked for quality and total cost "under matched conditions"; Table 7 and Appendix A.6 now report them.

2. "A plain LLM reranker ... matches or outperforms LIBRARIAN"

Row (c) is not a plain reranker and not an external baseline. It is LIBRARIAN's query planner, on live Europe PMC, over LIBRARIAN's candidate pool, with LIBRARIAN's LLM; only the output of the last stage is changed. Neither row has an index, so both support the claim of the paper. The comment also omits rows (f) and (g) of the same table: under the reranker's evidence budget, LIBRARIAN is 2.3 Citation F1 points above the reranker (row g, significant; one seed, 200 questions) and ties it on LitQA2 with 2.7x less text. In the default setting (row e) it ties on LitQA2 with 8x less text and has equal coverage on ScholarQA, with lower Citation F1.

On fielded queries we agree, and the revised paper states that their accuracy benefit is not established.

3. "Rather than a new retrieval method"

The paper does not claim a new retrieval method. W1 asked for "a substantial methodological advance or a sufficiently new empirical finding", and the reviewer’s comment addresses only the first. The findings are:

  • With corpus (237M passages) and generator fixed, dense embeddings add 1.7 and 0.6 points over BM25; an LLM second stage over a lexical index matches or exceeds the full dense pipeline.
  • The same quality holds on a live public search service, with no index built, hosted or refreshed.
  • GPT-5.4 calling the same API as an agent reaches 44.0% on LitQA2, and 72.5% through the layer at a tenth of the cost.

To our knowledge, no prior work reports these results: the closest studies (Hsu et al., 2026; Hu et al., 2026) build and host their own index and enrich every document offline. They are what a reader learns from this paper, and the informativeness rating of 1 ("I learned little or nothing new") does not take them into account.

Every experiment requested in the review is now in the paper. Q1 asked for quality, latency, and total cost against a dense or hybrid pipeline on the same corpus and backbone, and for the cost of the main configuration: Table 7 and Appendix A.5–A.6 report them. Q2 asked which design choice distinguishes LIBRARIAN from a reranking pipeline and what the combination establishes: Table 4 and Section 3.5 report them.

The reviewer guidelines ask reviewers to state what did and did not change their recommendation. The reviewer’s comment names the reranker row and fielded queries; it does not mention the result on Neu, the break-even, rows (f)–(g), or the direct-API agent. We respectfully request a re-evaluation in light of all these points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their thorough and constructive feedback. We are glad that the reviewers see "real practical value" in the design (R-Rqcw), "gains across several benchmarks and downstream agents" (R-cYur), and a control that "targets a central part of the method" (R-hdZp).

The concerns relate to (1) what is new beyond known components, (2) whether the gains are due to retrieval, and (3) completeness and cost. Each of these required a new set of ablations. We ran these comparisons, and the results are in the revised PDF (edits are highlighted).

What the paper contributes. A knowledge layer for scientific agents that retrieves from a search index built and maintained by a public service (Europe PMC), instead of a dense index built for the purpose. With no index of its own, it matches a dense pipeline on literature synthesis, and the same layer, with no task-specific index or training, serves agents for literature synthesis, factoid question answering, and claim verification.

What the new experiments show

  1. It matches dense retrieval (Reviewers cYur, Rqcw). On OpenScholar's own corpus and with the same generator, LIBRARIAN matches OpenScholar's dense retriever and reranker on Bio (74.8 against 74.2 Citation F1), exceeds it on Neu (77.2 against 73.0), and exceeds a BM25 + dense hybrid on both. On live Europe PMC, with no index at all, it scores 74.4 and 76.2. The dense pipeline instead holds 365 GB of embeddings in memory.
  2. The layer is needed (Reviewer Rqcw). GPT-5.4 calling the Europe PMC API itself, as an agent, answers 44.0% of LitQA2. Through LIBRARIAN, it answers 72.5% of the same questions, at a tenth of the cost.
  3. Where the gain comes from (Reviewers cYur, Rqcw). LLM query planning and LLM evidence selection take Europe PMC from 27.5% to 76.6% on LitQA2. LIBRARIAN matches multi-query BM25 with an equally strong LLM reranker while passing the agent 8x fewer tokens; on ScholarQA it draws on more papers (44 vs 12), with equal Prometheus coverage and lower Citation F1.
  4. The gain is due to retrieval (Reviewers Rqcw, hdZp). With corpus and generator fixed, LIBRARIAN improves Citation F1 over BM25 by 6.0 points on Bio and 8.7 on Neu, and by 8.3 and 8.2 when GPT-4o writes the answer from the same passages. With the agent, prompt, and verdict model fixed, the claim-verification agreement increases from 0.53 to 0.68. Inside ProClaim's own loop, it performs on par with PubMed + Semantic Scholar (0.57 against 0.58).
  5. The answers are more complete (Reviewer hdZp). On the same corpus, coverage exceeds BM25 (4.46 vs. 4.22), with about 8 more supported statements per answer, and is on par with the dense pipeline.
  6. LAB-Bench is now the control (Reviewer Rqcw). With the prompt matched, the gain we reported on sequence tasks comes from the answering instruction. We have corrected that claim and removed it from the abstract, introduction, and figures. With the same model, retrieval adds 60 points on LitQA2 (17.6% to 78.0%), where the answer is in the literature, and nothing on these biology workflow tasks, where it is not.
  7. Cost (Reviewers cYur, Rqcw). LIBRARIAN costs about $0.01 more per question than a dense pipeline and needs no index to build, host or refresh. With exact search over the embeddings, as in OpenScholar’s released code, owning a dense index pays off only above about 4 million questions per year on a CPU host, where a query takes tens of seconds, and above about 32 million for sub-second search on GPUs.

Conventions. Bio and Neu are the biomedicine (1,451 questions) and neuroscience (1,308) splits of ScholarQABench. "Significant" means that the 95% bootstrap interval of the paired difference over questions excludes zero (10,000 resamples); p-values are exact McNemar tests. Every new experiment uses GLM-5.3-Flash. The GLM-5 rows stay in the paper as submitted, with the GLM-5.3-Flash rows next to them: LIBRARIAN scores 74.4 / 76.2 on Bio / Neu with GLM-5.3-Flash (73.8 / 79.4 with GLM-5) and 74.0% on LitQA2 (78.0% with GLM-5).

We respectfully request a re-evaluation in light of these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.