- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Large Knowledge Model (LKM), a scalable and continuously extensible knowledge foundation for large-scale agentic science. LKM represents scientific papers as source-grounded reasoning graphs and further organizes them into three cross-paper views: Question, Workflow, and Evidence. The current system contains approximately 1.4 billion vectorized claims and supports incremental updates and agent-oriented retrieval. Experiments across multiple scientific retrieval and question-answering benchmarks show that LKM improves evidence discovery, citation reliability, and downstream scientific QA performance.
Critical strengths
-
The paper extends scientific knowledge representation beyond traditional document-level retrieval to fine-grained, traceable reasoning objects, explicitly preserving provenance and reasoning dependencies. This enables scientific agents not only to retrieve conclusions, but also to trace the supporting evidence and reasoning behind them.
-
LKM contains approximately 1.4 billion claims and nearly one million question families, while supporting incremental updates, high-concurrency retrieval, and multiple agent-facing interfaces. Its scale and engineering completeness make it particularly valuable as a practically deployable scientific knowledge infrastructure.
Critical weaknesses
-
LKM is positioned as a knowledge foundation for large-scale agentic science, yet the current evaluation focuses primarily on evidence retrieval, citation quality, and single-turn QA, without directly validating core capabilities such as multi-agent collaboration, long-term knowledge reuse, or incremental write-back. In my view, this provides limited empirical support for the paper’s central novelty.
-
Different benchmarks rely on different, largely single backbones, with no systematic cross-model replication under a unified setting and limited justification for the model choices. In addition, the quality of large-scale knowledge extraction is audited on only 100 papers, which is insufficient to fully establish the robustness of the overall system.
Questions for authors
-
Question, Workflow, and Evidence Landscapes are core components of LKM, yet their individual contributions across the different benchmarks remain unclear. Could the authors provide component ablations to quantify the contribution of each component?
-
Different benchmarks use different, largely single backbones. Could the authors clarify the rationale behind these model choices and replicate the experiments with additional backbones on each benchmark to assess cross-model generalization? It would also be helpful to consolidate the metrics, settings, and evaluation protocols in a dedicated section.
-
The current extraction-fidelity audit covers only 100 papers. Could the authors expand the human validation and report key details such as annotator expertise and number, annotation time and cost, evaluation prompts, and inter-annotator agreement to more thoroughly assess the reliability of a knowledge base containing 1.4 billion claims?
6 comments
W2. Both additional writers improved citation F1 on both ScholarQA subsets when given LKM graphs.
We appreciate this request. Testing a single backbone left open whether the graph effect was specific to GPT-4o, and we are glad to have checked. Both writers saw the same ten papers. Graphs covered the first three. We checked citations against common source passages.
| Writer | Subset (n) | Search + rerank | + Graph | Gain [95% CI] |
|---|---|---|---|---|
| Gemini 2.5 Flash | CS (100) | 62.02 | 65.01 | +2.99 [+0.64, +5.37] |
| DeepSeek-V4-Flash | CS (100) | 57.88 | 59.86 | +1.98 [−0.15, +4.08] |
| Gemini 2.5 Flash | Multi (108) | 66.93 | 70.77 | +3.84 [+1.67, +5.94] |
| DeepSeek-V4-Flash | Multi (108) | 62.88 | 66.41 | +3.53 [+1.54, +5.50] |
All four gains are positive, and three of the four intervals exclude zero. Gemini has no role in building LKM yet shows the larger gains, so the benefit does not depend on the writer sharing a model family with the extractor. Our takeaway is that the citation benefit of reasoning graphs carries over across model families.
QA accuracy (%) on matched questions.
| Benchmark (n) | Answerer | No tool | Commercial search | LKM |
|---|---|---|---|---|
| ChemBench (899) | GPT-4o | 55.39 | 56.95 | 57.51 |
| DeepSeek-V4-Pro | 57.40 | 58.73 | 59.07 | |
| Qwen 3.6 Plus | 70.63 | 68.63 | 68.74 | |
| PubMedQA (500) | GPT-4o | 56.80 | 55.00 | 55.40 |
| DeepSeek-V4-Pro | 64.60 | 64.60 | 67.40 | |
| Qwen 3.6 Plus | 80.40 | 79.80 | 80.80 | |
| SciBench (580) | GPT-4o | 31.38 | 29.48 | 31.03 |
| DeepSeek-V4-Pro | 57.41 | 52.07 | 51.90 | |
| Qwen 3.6 Plus | 77.93 | 80.00 | 78.28 |
LKM ranked first with two answerers each on ChemBench and PubMedQA. Its paired advantages over commercial search in those four rows ranged from 0.33 to 2.80 points.
The QA picture is more modest than the citation results, and we think it is worth explaining why. These benchmarks leave retrieval relatively little to add: strong answerers already solve many questions from parametric knowledge, every PubMedQA question already includes the study abstract, and much of SciBench consists of textbook calculations that research papers rarely spell out. Where retrieval does help, LKM is usually the better source, but the gains depend on the answering model. We present these results in the revision with that caveat.
Q1. Measuring the individual contribution of the Question, Workflow, and Evidence Landscapes to agent performance remains future work.
Thank you for this question; it made us realize that the manuscript did not make clear which components each benchmark actually exercises. The Section 5 benchmarks use paper-level retrieval and reasoning graphs. The ScholarQA ablation therefore measures what a writer gains from seeing the reasoning graphs for the same retrieved papers.
The landscapes organize these graphs across papers into units that applications consume. Workflow families group papers sharing a procedure and keep the paper instances needed to instantiate a task; they have supplied approximately 17,000 candidate verifiable tasks. Question families group related problems with their sources; researchers used them to trace original arguments and check later literature when formulating approximately 50 open or semi-open problems. The Evidence Engine retrieves supporting, opposing, and conditional findings for a claim and serves a deployed manuscript-review agent. In our new SciConBench study, the LKM agent called the Evidence Engine alongside search, claim tracing, and paper graphs, so that study measures the toolset as a whole.
We think the honest answer is that a landscape’s contribution has to be measured on the outcomes of the application that uses it, for example the quality of tasks built with and without workflow families, rather than on a QA benchmark that never calls it. We have not run such controlled comparisons yet. We stated in Section 5 which components each benchmark uses, described the applications in Section 4.2, and noted in the limitations that the landscapes’ individual contributions remain future work.
Q2. Each model was chosen for its role in a benchmark. Each original benchmark with a generating model now has three backbones, and a new protocol appendix consolidates the settings.
Thank you for this suggestion; a single place for settings and protocols is clearly better for readers. SciFact-Open and PaSaMaster score retrieved papers directly and involve no answering model. For the original experiments, we used GPT-4o in ScholarQA because it is OpenScholar’s writer, keeping our scores comparable with published baselines. We used GPT-5.4 in QA to test whether retrieval still helps when a strong answerer is held fixed. We used DeepSeek-V4-Flash for extraction because corpus-scale cost dominates (CNY 0.55 per paper above). The added writers cover a model family with no role in LKM (Gemini) and an open-weight model (DeepSeek-V4-Flash). We selected DeepSeek-V4-Pro for QA on a separate 100-question ChemBench development set, and the SciConBench agent on 12 development questions.
| Benchmark | Models | Evidence per question | Metric (judge) |
|---|---|---|---|
| ScholarQA-CS / Multi | GPT-4o, Gemini 2.5 Flash, DeepSeek-V4-Flash | Top-10 papers; graphs for top 3 | AttrScore citation F1; Overall (GPT-4o, 3 runs) / Prometheus |
| ChemBench, PubMedQA, SciBench | GPT-4o, DeepSeek-V4-Pro, Qwen 3.6 Plus | Up to 5 papers | Exact match; 5% numeric tolerance |
| SciConBench | Qwen 3.6 Plus | Up to 12 tool calls | Atomic-fact recall / F1 (GPT-4o) |
Each condition produced one answer per question. All intervals are paired over questions with 10,000 bootstrap resamples. The new runs rank candidates by LKM’s rerank score and use a stratified ChemBench test split, so we report them separately from Figure 5 and Table 3. We added a new appendix that consolidates this protocol and reports the complete per-model results, and Section 5 now points to it.
Thank you very much for your detailed and thoughtful review. We are glad that you recognized how LKM preserves provenance and reasoning dependencies, and we appreciate your highlighting the scale, incremental updates, and concurrent retrieval of the deployed system. Your questions about component contributions, cross-model generalization, and audit reliability were very crucial, and answering them has made the paper more rigorous.
In response, we measured production updates, replicated the writing and QA experiments across models, and expanded the fidelity audit to 300 papers. We also detail each landscape’s applications and each task’s model choices and evaluation protocol. We address each point below, and would be happy to provide any further details.
W1. The production service makes new papers searchable one graph at a time. Long-term reuse across agents and write-back of agent results remain prospective uses.
Thank you for this comment. We agree that the manuscript did not separate clearly enough what we have evaluated from what LKM is designed to enable, and we made that distinction explicit in the revision. What we can show today is incremental update. Each new paper updates only the objects it introduces or matches, so later queries can reach them without rebuilding the corpus. On November 12, the production service recorded 2,618 completed ingestions and 18 failures. For 2,933 papers with aligned stage logs, median time from OCR completion to extraction completion was 85 minutes, followed by a median 45 seconds to graph insertion. Graphs typically become searchable within seconds of insertion. In a sample of 12 papers stratified by length, LLM extraction averaged CNY 0.55 per paper, excluding OCR and embedding.
In plain terms, once a new paper has been through OCR, it typically becomes usable by agents within about an hour and a half, at an extraction cost of well under one yuan per paper. Multi-agent reuse and write-back are what this capability is meant to enable, but we have not yet evaluated them, and we added this statement to the introduction and limitations.