- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LIBRARIAN, which expands a natural language question into complementary literature queries, locates passages in retrieved papers, and uses a language model to select and rank citable sentences. The authors connect it to agents for literature synthesis, scientific claim verification, open form question answering, and biology tasks, then compare their performance.
Critical strengths
The “Semantic evidence selection” ablation compares semantic selection with lexical ranking over the same candidate passage pool and evidence length limit, and finds clear differences in both open form question answering and literature synthesis. This control targets a central part of the method and gives more direct evidence for the value of fine grained selection than end to end scores alone.
Critical weaknesses
The claim verification results table compares the original ProClaim system with ProClaim using LIBRARIAN, and the main text interprets the higher agreement as better access to knowledge through retrieval. Yet the appendix's ProClaim implementation details explain that adaptation also changes fact extraction from reading full papers to labeling selected evidence directly, and replaces the original sufficiency classifier with a language model alternative. The original result is also taken from prior work. These changes could themselves affect which facts are selected, whether retrieval continues, and the final verdict. The comparison therefore does not isolate the retrieval layer's contribution or establish the claimed causal interpretation for this task.
The literature synthesis results table mainly reports citation support for the biomedical and neuroscience tasks. That measure checks whether cited sources support the statements the agent wrote; it cannot tell whether important content was omitted. In the “Full-text access” appendix ablation, restricting evidence to abstracts changes this measure only slightly despite the substantial loss of full text, and the authors acknowledge that it reflects consistency between claims and citations more than answer quality. A separate coverage related score on the multidisciplinary subset does not directly validate the gains on the main tasks or their fixed corpus comparison. Thus, the central interpretation that literature synthesis quality improves still lacks a measure aligned with completeness.
Questions for authors
Could you provide or identify a comparison that keeps ProClaim's fact extraction and sufficiency decision procedure fixed while changing only the retrieval layer, so the retrieval benefit can be separated from the downstream workflow changes?
Could you assess coverage or completeness of key facts for the main literature synthesis tasks under the same corpus and generation settings, to show whether the gain in citation support also means more complete synthesis?
4 comments
Kind Reminder: Discussion Deadline Approaching
Dear Reviewer hdZp,
Thank you again for your time and the detailed, constructive feedback you provided on our paper.
As the discussion period deadline is approaching, we wanted to follow up. We have submitted a detailed rebuttal addressing the concerns you raised, including additional experiments and analyses. To summarize, in our response, we have:
- Provided a controlled comparison on ProClaim, keeping fact extraction and the sufficiency decision fixed while varying only the retriever (Q1, Tables 3 and 14).
- Conducted additional completeness evaluations on the main synthesis tasks, keeping the corpus and generator fixed to ensure a fair comparison (Q2, Table 13, Appendix C.2).
Both points you raised are now covered by a controlled comparison.
We respectfully ask that you reconsider your rating in light of these results, and we welcome any further questions before the imminent deadline.
Thank you for your response. It addressed my concerns, and I have decided to increase the score.
We thank the reviewer for reconsidering the paper and for raising the score. We are glad that the two controls you asked for, the retriever isolated in ProClaim and completeness on the main synthesis tasks, addressed your concerns.
Since no concerns remain open, we would be grateful if you would consider whether the paper now merits a clear acceptance. If anything still holds you back, please let us know, and we will address it before the discussion closes.
Response to Reviewer hdZp
We thank the reviewer. Each of the two observations is answered below with a controlled comparison.
W1 and Q1. Isolating the retrieval layer in claim verification
The cleanest test of the retrieval layer is the Verifier Agent, where the agent, prompt, and verdict model are identical and only the retriever changes: LIBRARIAN raises agreement with expert consensus from 0.53 to 0.68 across all 406 labeled claims (419 in the submission were with an old version of the released dataset).
We also ran the comparison you ask for inside ProClaim. The last row keeps ProClaim's fact extraction and sufficiency decision unchanged: LIBRARIAN only finds the papers, and ProClaim's own sub-agent reads their full text. The two retrievers do not differ significantly. The lower figure reflects our one-hour limit per claim: reading whole papers is slower and this arm ran on a slower GPU, so 39 claims reached the limit against 25, and they count as wrong. On the 52 claims that every arm answered in time, agreement is the same. On all 406 claims (second row) the retrievers are on par. So inside a loop designed around PubMed and Semantic Scholar, LIBRARIAN replaces both at no loss in agreement, and in a simple agent it raises agreement from 0.53 to 0.68.
As you correctly noted, rows 0.79 and 0.84 of Table 3 also differ in fact extraction and in the sufficiency classifier, and that 0.79 is taken from the Proclaim pipeline, so the comparison does not isolate the effect of the retriever. What it does show is a consequence of the evidence LIBRARIAN returns: because that evidence is already selected for relevance to the claim, it can be processed in a much simpler way than in the original agent, and this is enough to reach 0.84. The revised table labels the two rows accordingly, and the text presents the Verifier Agent as the retrieval result. In every arm above, the sufficiency decision is made by the LLM classifier that ProClaim provides, since its MLP needs impact factor, citation counts, and full-text features that LIBRARIAN's evidence does not carry.
W2 and Q2. Completeness on the main synthesis tasks
We measured it on the main tasks, with the corpus (OpenScholar's) and the generator (GLM-5.3-Flash) held fixed. LIBRARIAN's answers score higher than BM25's on coverage and organization, contain more supported statements from more sources, and are on par with the dense pipeline's; relevance is the same for all three (Table 13, Appendix C.2).
† significantly different from LIBRARIAN. Prometheus rows and word counts: 100 Bio and 100 Neu questions. Statement rows: full splits.
Each answer contains about 8 more supported statements, drawn from 10 more sources, and a larger share of its citation-worthy statements is supported, a ratio that does not depend on answer length. In a pairwise comparison of key-fact coverage (GPT-4o judge, both orders), LIBRARIAN's answer is preferred to BM25's on 136 of the 200 questions, tied on 44 and not preferred on 20.
LIBRARIAN's answers are also longer, and the gain in Prometheus coverage is concentrated in the questions where they are longer. We therefore present the result as more strongly supported by more sources, and Sec. 3.2 now says explicitly that Citation F1 does not measure completeness. Against the dense pipeline, coverage is the same.
To Reviewer hdZp: Both points are now covered by a controlled comparison: the retriever is isolated in claim verification, and completeness is measured on the main tasks with the corpus and generator fixed. We respectfully request a re-evaluation in light of these results.