acceptodds
Under review as a conference paper at ICLR 2027

Large Knowledge Model: A Knowledge Foundation for Agentic Science at Scale

Abstract

Agentic science envisions many autonomous agents investigating concurrently while building on a shared, evolving body of scientific knowledge. This requires a knowledge foundation that supports high-concurrency access, preserves traceable and reusable reasoning, and grows incrementally. We propose the Large Knowledge Model (LKM), a growing, agent-native knowledge foundation that provides a general representation of scientific knowledge across disciplines. LKM organizes the scientific literature into reasoning graphs, with claims as the core nodes and associated reasoning chains that make explicit how premises and evidence support conclusions. These source-grounded objects are persistent and addressable; cross-paper links organize them into aligned question, workflow, and evidence views. Newly extracted papers extend the foundation incrementally while preserving existing object identities. Building on this foundation, we develop an agent-native, reasoning-aware scientific retrieval system that retrieves claims together with their reasoning chains and sources, enabling agents to inspect and reuse the evidence underlying scientific conclusions. Across benchmarks, agents using LKM retrieve more evidence, cite more faithfully, and answer scientific questions more accurately: LKM nearly doubles the known supporting and contradicting evidence retrieved on SciFact-Open (818 versus 443 claim–paper pairs), reasoning graphs raise citation F1 on ScholarQABench by more than 5 points over the same retrieved papers, and LKM retrieval improves a fixed answering model by 9.3, 4.2, and 14.7 points over no retrieval on ChemBench, PubMedQA, and SciBench. LKM lays the foundation for a scientific ecosystem in which AI scientists not only recall accumulated knowledge but also extend it, returning new questions, workflows, and evidence to a memory that every subsequent investigation can build on.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Reasoning Landscape, Reasoning Graphs, Scientific Knowledge Representation, Scientific Information Retrieval, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UZoJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Large Knowledge Model (LKM), a scalable and continuously extensible knowledge foundation for large-scale agentic science. LKM represents scientific papers as source-grounded reasoning graphs and further organizes them into three cross-paper views: Question, Workflow, and Evidence. The current system contains approximately 1.4 billion vectorized claims and supports incremental updates and agent-oriented retrieval. Experiments across multiple scientific retrieval and question-answering benchmarks show that LKM improves evidence discovery, citation reliability, and downstream scientific QA performance.

Critical strengths
  1. The paper extends scientific knowledge representation beyond traditional document-level retrieval to fine-grained, traceable reasoning objects, explicitly preserving provenance and reasoning dependencies. This enables scientific agents not only to retrieve conclusions, but also to trace the supporting evidence and reasoning behind them.

  2. LKM contains approximately 1.4 billion claims and nearly one million question families, while supporting incremental updates, high-concurrency retrieval, and multiple agent-facing interfaces. Its scale and engineering completeness make it particularly valuable as a practically deployable scientific knowledge infrastructure.

Critical weaknesses
  1. LKM is positioned as a knowledge foundation for large-scale agentic science, yet the current evaluation focuses primarily on evidence retrieval, citation quality, and single-turn QA, without directly validating core capabilities such as multi-agent collaboration, long-term knowledge reuse, or incremental write-back. In my view, this provides limited empirical support for the paper’s central novelty.

  2. Different benchmarks rely on different, largely single backbones, with no systematic cross-model replication under a unified setting and limited justification for the model choices. In addition, the quality of large-scale knowledge extraction is audited on only 100 papers, which is insufficient to fully establish the robustness of the overall system.

Questions for authors
  1. Question, Workflow, and Evidence Landscapes are core components of LKM, yet their individual contributions across the different benchmarks remain unclear. Could the authors provide component ablations to quantify the contribution of each component?

  2. Different benchmarks use different, largely single backbones. Could the authors clarify the rationale behind these model choices and replicate the experiments with additional backbones on each benchmark to assess cross-model generalization? It would also be helpful to consolidate the metrics, settings, and evaluation protocols in a dedicated section.

  3. The current extraction-fidelity audit covers only 100 papers. Could the authors expand the human validation and report key details such as annotator expertise and number, annotation time and cost, evaluation prompts, and inter-annotator agreement to more thoroughly assess the reliability of a knowledge base containing 1.4 billion claims?

6 comments

Thank you very much for your detailed and thoughtful review. We are glad that you recognized how LKM preserves provenance and reasoning dependencies, and we appreciate your highlighting the scale, incremental updates, and concurrent retrieval of the deployed system. Your questions about component contributions, cross-model generalization, and audit reliability were very crucial, and answering them has made the paper more rigorous.

In response, we measured production updates, replicated the writing and QA experiments across models, and expanded the fidelity audit to 300 papers. We also detail each landscape’s applications and each task’s model choices and evaluation protocol. We address each point below, and would be happy to provide any further details.

W1. The production service makes new papers searchable one graph at a time. Long-term reuse across agents and write-back of agent results remain prospective uses.

Thank you for this comment. We agree that the manuscript did not separate clearly enough what we have evaluated from what LKM is designed to enable, and we made that distinction explicit in the revision. What we can show today is incremental update. Each new paper updates only the objects it introduces or matches, so later queries can reach them without rebuilding the corpus. On November 12, the production service recorded 2,618 completed ingestions and 18 failures. For 2,933 papers with aligned stage logs, median time from OCR completion to extraction completion was 85 minutes, followed by a median 45 seconds to graph insertion. Graphs typically become searchable within seconds of insertion. In a sample of 12 papers stratified by length, LLM extraction averaged CNY 0.55 per paper, excluding OCR and embedding.

In plain terms, once a new paper has been through OCR, it typically becomes usable by agents within about an hour and a half, at an extraction cost of well under one yuan per paper. Multi-agent reuse and write-back are what this capability is meant to enable, but we have not yet evaluated them, and we added this statement to the introduction and limitations.

W2. Both additional writers improved citation F1 on both ScholarQA subsets when given LKM graphs.

We appreciate this request. Testing a single backbone left open whether the graph effect was specific to GPT-4o, and we are glad to have checked. Both writers saw the same ten papers. Graphs covered the first three. We checked citations against common source passages.

WriterSubset (n)Search + rerank+ GraphGain [95% CI]
Gemini 2.5 FlashCS (100)62.0265.01+2.99 [+0.64, +5.37]
DeepSeek-V4-FlashCS (100)57.8859.86+1.98 [−0.15, +4.08]
Gemini 2.5 FlashMulti (108)66.9370.77+3.84 [+1.67, +5.94]
DeepSeek-V4-FlashMulti (108)62.8866.41+3.53 [+1.54, +5.50]

All four gains are positive, and three of the four intervals exclude zero. Gemini has no role in building LKM yet shows the larger gains, so the benefit does not depend on the writer sharing a model family with the extractor. Our takeaway is that the citation benefit of reasoning graphs carries over across model families.

QA accuracy (%) on matched questions.

Benchmark (n)AnswererNo toolCommercial searchLKM
ChemBench (899)GPT-4o55.3956.9557.51
DeepSeek-V4-Pro57.4058.7359.07
Qwen 3.6 Plus70.6368.6368.74
PubMedQA (500)GPT-4o56.8055.0055.40
DeepSeek-V4-Pro64.6064.6067.40
Qwen 3.6 Plus80.4079.8080.80
SciBench (580)GPT-4o31.3829.4831.03
DeepSeek-V4-Pro57.4152.0751.90
Qwen 3.6 Plus77.9380.0078.28

LKM ranked first with two answerers each on ChemBench and PubMedQA. Its paired advantages over commercial search in those four rows ranged from 0.33 to 2.80 points.

The QA picture is more modest than the citation results, and we think it is worth explaining why. These benchmarks leave retrieval relatively little to add: strong answerers already solve many questions from parametric knowledge, every PubMedQA question already includes the study abstract, and much of SciBench consists of textbook calculations that research papers rarely spell out. Where retrieval does help, LKM is usually the better source, but the gains depend on the answering model. We present these results in the revision with that caveat.

Q1. Measuring the individual contribution of the Question, Workflow, and Evidence Landscapes to agent performance remains future work.

Thank you for this question; it made us realize that the manuscript did not make clear which components each benchmark actually exercises. The Section 5 benchmarks use paper-level retrieval and reasoning graphs. The ScholarQA ablation therefore measures what a writer gains from seeing the reasoning graphs for the same retrieved papers.

The landscapes organize these graphs across papers into units that applications consume. Workflow families group papers sharing a procedure and keep the paper instances needed to instantiate a task; they have supplied approximately 17,000 candidate verifiable tasks. Question families group related problems with their sources; researchers used them to trace original arguments and check later literature when formulating approximately 50 open or semi-open problems. The Evidence Engine retrieves supporting, opposing, and conditional findings for a claim and serves a deployed manuscript-review agent. In our new SciConBench study, the LKM agent called the Evidence Engine alongside search, claim tracing, and paper graphs, so that study measures the toolset as a whole.

We think the honest answer is that a landscape’s contribution has to be measured on the outcomes of the application that uses it, for example the quality of tasks built with and without workflow families, rather than on a QA benchmark that never calls it. We have not run such controlled comparisons yet. We stated in Section 5 which components each benchmark uses, described the applications in Section 4.2, and noted in the limitations that the landscapes’ individual contributions remain future work.


Q2. Each model was chosen for its role in a benchmark. Each original benchmark with a generating model now has three backbones, and a new protocol appendix consolidates the settings.

Thank you for this suggestion; a single place for settings and protocols is clearly better for readers. SciFact-Open and PaSaMaster score retrieved papers directly and involve no answering model. For the original experiments, we used GPT-4o in ScholarQA because it is OpenScholar’s writer, keeping our scores comparable with published baselines. We used GPT-5.4 in QA to test whether retrieval still helps when a strong answerer is held fixed. We used DeepSeek-V4-Flash for extraction because corpus-scale cost dominates (CNY 0.55 per paper above). The added writers cover a model family with no role in LKM (Gemini) and an open-weight model (DeepSeek-V4-Flash). We selected DeepSeek-V4-Pro for QA on a separate 100-question ChemBench development set, and the SciConBench agent on 12 development questions.

BenchmarkModelsEvidence per questionMetric (judge)
ScholarQA-CS / MultiGPT-4o, Gemini 2.5 Flash, DeepSeek-V4-FlashTop-10 papers; graphs for top 3AttrScore citation F1; Overall (GPT-4o, 3 runs) / Prometheus
ChemBench, PubMedQA, SciBenchGPT-4o, DeepSeek-V4-Pro, Qwen 3.6 PlusUp to 5 papersExact match; 5% numeric tolerance
SciConBenchQwen 3.6 PlusUp to 12 tool callsAtomic-fact recall / F1 (GPT-4o)

Each condition produced one answer per question. All intervals are paired over questions with 10,000 bootstrap resamples. The new runs rank candidates by LKM’s rerank score and use a stratified ChemBench test split, so we report them separately from Figure 5 and Table 3. We added a new appendix that consolidates this protocol and reports the complete per-model results, and Section 5 now points to it.

Reviewer_CFfgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LKM, a large-scale scientific knowledge system that extracts claims, reasoning chains, and sources from research papers. It organizes this information into question, workflow, and evidence views, with APIs that let agents access it. The experiments evaluate evidence retrieval, citation quality, and scientific question answering.

Critical strengths
  • The benchmarks are well aligned with the system’s retrieval and QA functions. They provide useful evaluations of evidence coverage, citation quality, and answer accuracy.

  • The scale is impressive, with about 1.4 billion claims and a deployed retrieval service. The API operations are clearly organized. This could be a useful product for researchers and scientific agents.

Critical weaknesses
  • The paper spends too much space selling concepts such as the Question, Workflow, and Evidence Landscapes, but I do not see clearly what these concepts are useful for. I would find a more concrete discussion of the system design more informative. The system could be a useful product, but the claims about accelerating science seem overstated.

  • The current benchmarks are reasonable for retrieval and QA, but they do not show how the system helps agents do scientific research. An evaluation like SciConBench [1] could test whether it improves scientific evidence synthesis beyond finding relevant papers and answering questions.

  • The extraction details are not clear enough, especially how the system checks claims and reasoning links. For a system at this scale, I would like to see more explicit, machine-checkable rules, along the lines of the reasoning language in Gaia [2]. This would not guarantee that every claim is correct, but could make the reasoning easier to inspect and debug. My concern is that even a small error rate could become difficult to manage, and it is unclear how to trace and correct incorrect claims and their dependent links.

[1] Jung et al. Can AI Agents Synthesize Scientific Conclusions? NeurIPS 2026, Evaluations and Datasets Track. SciConBench

[2] Gaia: Formal Language for Natural Science. GitHub

Questions for authors
  • On SciFact-Open, what are the results using only the original human annotations? Did humans check a sample of the newly added positive pairs?
6 comments

Thank you very much for your thoughtful review. We are grateful that you recognized how closely our benchmarks match LKM’s retrieval and QA functions, as well as the scale of the deployed service and the clarity of its APIs. Your suggestions, especially the pointer to SciConBench, led directly to new experiments that we think make the paper stronger.

Following your SciConBench suggestion, we tested the same agent with four tool settings on two independent sets of 100 questions. We also describe how researchers use each landscape, show how to trace a disputed claim to its source and dependent conclusions, and report the human adjudication of added SciFact-Open labels. We respond to each point below, and we would be glad to answer any further questions.

W1. Workflow families supply procedures for verifiable tasks, question families help researchers formulate open problems, and the Evidence Engine checks manuscript claims against the literature.

Thank you for this comment. We agree that the original manuscript described the landscapes more abstractly than it should have, and it was not easy for a reader to see what they are actually used for. A workflow family groups papers that use a common procedure and retains the paper instances needed to instantiate a task. These families have supplied approximately 17,000 candidate verifiable tasks. A question family groups related problems and their sources. Researchers used these families to trace original arguments and check subsequent literature when formulating approximately 50 open or semi-open research problems. Both collections are used for agent evaluation and training. The Evidence Engine serves a deployed manuscript-review agent that compares a claim with supporting, opposing, and conditional findings.

In short, each landscape already has a concrete consumer: task construction, problem formulation, and manuscript review. We redrew Figure 3 to show where these applications enter the system. Search can start from an individual paper graph or a family and expand the result into reasoning steps and source papers. We added a description of these applications to Section 4.2 and noted in Section 4.3 that search reaches both paper graphs and families.

W2. On both SciConBench samples, the agent covered more reference facts with LKM than with commercial paper search or web search. Its F1 relative to commercial search varied across samples.

We are grateful for the suggestion to use SciConBench; it tests an agent’s use of the literature more directly than our original benchmarks did. We held the Qwen 3.6 Plus agent fixed across four conditions on two disjoint sets of 100 review questions. Each tool-enabled agent could make up to 12 calls per question. We used SciConBench’s atomic-fact scoring procedure with GPT-4o as the substitute judge. Each cell gives mean factual recall / F1, with F1 calculated per question.

Qwen 3.6 Plus conditionFirst 100, recall / F1Independent 100, recall / F1
No tool36.96 / 28.4440.93 / 33.18
Commercial paper search38.52 / 24.9838.95 / 26.40
Web search36.16 / 29.6437.28 / 30.04
LKM38.58 / 27.4040.07 / 26.13

LKM minus commercial search was +2.42 F1 points on the first set (paired 95% CI −0.26 to +5.15) and −0.27 on the independent set [−2.96, +2.44]. On the first set it obtained source passages for all 100 questions, followed claim traces 67 times, and opened paper graphs 20 times, so the agent did make real use of the reasoning-level tools. On the independent set, LKM recovered more reference facts than web search, while web search produced a higher proportion of supported facts (31.13 versus 23.23 precision).

We want to be candid that this is a mixed result. LKM helps the agent find and inspect more of the relevant evidence, but finding more evidence does not automatically produce a tighter final answer, and on F1 no tool setting clearly beats answering without tools. We read this as separating two abilities: finding and inspecting evidence, and selecting the facts that belong in the final conclusion. LKM addresses the first; the second depends largely on the agent. We summarized these results in Section 5.3 and added them in full, with this interpretation, to a new appendix.

W3. An auditor can trace a disputed claim to its source, inspect the premises used to derive it, and identify downstream conclusions that rely on it.

Thank you for asking about auditability; it is central to why we built LKM around explicit reasoning structure. An inference factor records an ordered set of premises, a reasoning chain, and a conclusion. The typed edges premise_of and concludes tell an auditor which claims a factor consumes and which claim it produces. For example, in Figure 2, conclusions 2, 4, 5, and 6 are premises for the inference yielding conclusion 3. Conclusions 8, 11, and 12 then use conclusion 3. An auditor querying a suspected error in conclusion 3 can therefore locate the source argument and identify all three downstream conclusions for review.

The schema makes edge direction and endpoint type machine-checkable. We added to Section 3 how these checks locate dependent objects for review. In the expanded 300-paper audit, 560 quotations with source-text line numbers document the 162 extraction errors. Under our protocol, each retained issue identifies the extracted wording and source passage, explains the material change in meaning, and is checked again against the paper. For instance, one source gave incompatible F1 values in its text and table. The audit recorded the inconsistency at its source so an auditor can see which value the extraction followed. In practice, this means a disputed claim is never a dead end: one can always walk back to the text it came from and forward to everything that depends on it.

Reviewer_7mr2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LKM, which organizes scientific papers into source-linked claims, inferential relations, and reasoning chains, with incremental updates and retrieval interfaces. The authors report approximately 1.4 billion vectorized claims and three organizational views covering questions, workflows, and evidence. Experiments evaluate evidence retrieval, citation quality, and scientific question answering; adding reasoning graphs for the same retrieved papers increases citation F1 by 5.3 and 5.1 percentage points on the two ScholarQABench subsets.

Critical strengths

Clear ablation for targeted representation. The ScholarQABench comparison holds candidate papers, reranking, and the paper budget fixed while testing the effect of additionally providing reasoning graphs, directly addressing the proposed representation. Complete Evaluation spans retrieval and answering. The experiments separately report evidence recall, citation quality, paper retrieval, and QA with a fixed answering model, providing observations at multiple stages of system use.

Critical weaknesses

The added reference labels need independent calibration. Section 5.1 already requires agreement between DeepSeek and MultiVerS for new positives and gives original human annotations precedence. These safeguards should be credited. What remains unreported is human validation of the added labels and a separate breakdown on the original 460 human-positive pairs versus the expanded 1,125-pair set. Please provide those breakdowns and a blinded audit of a sample of added pairs to assess how label expansion affects the reported retrieval gain. The fidelity audit needs an estimate of screening misses. The 0.97% figure is explicitly a confirmed hallucination rate from an LLM screen followed by expert review of flagged objects, and the paper acknowledges that omissions are not assessed. The unresolved issue is the screen’s miss rate: random expert checks of unflagged objects and reasoning relations are not reported. Please add random expert checks of unflagged material and report their findings alongside the existing audit.

Questions for authors
  1. The scale of LKM is claimed to be 1.4billion vectorized public claims. What about the actual scale of source corpus? how many papers and how papers are collected or filtered. And how many subjects does the corpus cover?

  2. In Section 5.2, LKM Search, LKM Search with reranking, and LKM Graph with reranking are compared. is there more details about experiments. And I think contexts from retrieved papers can be compared with LKM Graph, to show effectiveness extracted graph. I think 10-paper budge is acceptable for frontier LLMs.

  3. It seems that only texts are involved in LKM, what about scientific elements such as formulas, images or tables?

5 comments

Thank you very much for your careful and constructive review. We are glad that you found value in our matched-paper graph ablation and in our choice to evaluate evidence retrieval, citation quality, and answer accuracy separately. Your questions led us to run several new analyses, and we believe the paper is now both clearer and more candid about what LKM does well and where it still falls short.

In response, we audited the added SciFact-Open labels, expanded the source checks to 300 papers, and compared graphs with passages on all 208 ScholarQABench questions. We also clarify the corpus and the treatment of equations, figures, and tables. We address each point below. Your comments have substantially improved the paper, and we would be happy to clarify anything further. 

W1. LKM recovered fewer original human-labeled pairs than the released retriever, mainly because part of that gold was collected from the released retriever’s own candidates and because LKM ranks against a much larger corpus. LKM also surfaced 662 additional model-consensus positives, 441 supporting and 221 contradicting.

Thank you for pressing on this point. It is a fair concern, and looking into it closely taught us a good deal about how the benchmark was built. Each system returned 50 papers per claim. We split the original gold by how SciFact-Open collected it. Citation pairs come from SciFact-Orig citation links and do not depend on any retriever. Pooled pairs were annotated from model predictions over the released retriever’s candidates. The last column uses LKM to retrieve and rerank among the top 1,000 candidates from the SciFact-Open corpus to separate out the difference caused by corpus size.

Positive pairs (n)Released retrieverLKM, open searchLKM rerank, benchmark corpus
Original, citation links (209)187 (89.5%)125 (59.8%)177 (84.7%)
Original, pooled (251)251 (100%)31 (12.4%)190 (75.7%)
Original total (460)438 (95.2%)156 (33.9%)367 (79.8%)
Added model consensus (665)5662—
Combined (1,125)443 (39.4%)818 (72.7%)—

Three observations explain the gap on original gold. First, pooling credits the system that built the pool: all 251 pooled pairs lie in the released retriever’s top 50, and papers outside its candidates were never judged. On retriever-independent citation pairs, the gap is 29.7 points, against 87.6 on pooled pairs. Second, coverage explains little, since LKM indexes 425 of the 460 gold papers. Third, LKM ranks these papers against a much larger corpus. The two systems’ top-50 lists share 364 of 27,533 claim–paper pairs (1.3%), and 15.6% of LKM’s results were published after 2020, whereas all of the released retriever’s candidates predate 2021. In other words, the two systems are largely looking at different parts of the literature.

When candidates come from the benchmark corpus, LKM reranking raises BM25’s original-gold hits from 350 to 367. On citation pairs whose papers LKM indexes, it reaches 90.8% versus 89.2% for the released retriever. In a stratified audit by internal domain experts of 200 LKM-added evidence pairs, adjudication retained 130/133 SUPPORT and 64/67 CONTRADICT labels, so nearly all of the sampled added evidence held up under review.

Put simply, the low original-gold recall says less about LKM’s ability to rank known evidence than about two facts: more than half of the original gold was collected from the released retriever’s own output, and LKM searches a far larger and more recent literature. When both systems search the same corpus, LKM ranks the retriever-independent evidence about as well as the released retriever. We nonetheless agree that recovering known evidence in open search is something LKM should do better, and the revised Section 5.1 now states plainly that the released retriever recovers more of the original pairs. We summarized this breakdown in Section 5.1 and added the full table to a new appendix, so readers can distinguish discovery of additional evidence from recovery of known evidence.

W2. Direct source checks across 300 papers identified 162 extraction errors. Estimating the original screen’s miss rate requires the screen decision for each checked item.

We appreciate this suggestion and performed an additional round of random checks of unflagged material. We added 25 randomly sampled papers per field across eight fields. The review protocol required reading each new paper in full, checking every conclusion and reasoning step against the available text regardless of earlier flags, and verifying each proposed error a second time. The two batches contributed 1,392 and 2,862 checked items. An error had to change a main conclusion or its key evidence or argument.

Paper sampleErrors introduced by extractionAll extracted itemsErrors / all items
Original 100, prior records473,9041.20%
Additional 2001158,0891.42%
Combined16211,9931.35%

The 162 affected items comprise 43 conclusions and 119 reasoning chains. Among the 4,254 conclusions and chains checked directly, 162 (3.81%) contained a confirmed error, roughly one in 26 items. Errors are more common in reasoning chains than in conclusions, which matches our intuition: a chain has more steps in which a comparison or condition can drift from the source. The 1.35% figure uses all 11,993 extracted items as its denominator, including premises and highlights. The audit records 560 line-referenced quotations for the extraction errors.

We want to be direct about one limitation. Because the new batch was checked without reference to the earlier LLM screen, this audit does not tell us that screen’s miss rate. What it does provide is an error estimate that no longer depends on what a screen happened to flag, which we think addresses the underlying concern more directly.

Q1. The source corpus contains tens of millions of papers, among which about 40 million had been processed from full text and about 70 million from abstracts.

Thank you for asking; the manuscript should have been clearer about this. Available PDFs are processed with OCR and then extracted into conclusions, premises, and reasoning chains. Records are deduplicated by DOI, title, and content hash. An early screening stage removes non-research documents, such as conference abstracts and errata, before extraction, so that expensive extraction is not spent on material that carries no research argument.

The corpus covers mathematics, physics, chemistry, astronomy, earth and life sciences, engineering, information science, and environmental science. An internal audit across mathematics, biology, physics, chemistry, engineering, and the social sciences found that coverage varies by field and publication year. We added these details to Section 4.1 so that readers know what the 1.4 billion claims are drawn from.

Comments on the paper

1 comment

We sincerely thank all three reviewers for their careful and constructive reviews. We are encouraged that the reviewers found value in LKM's preservation of provenance and reasoning dependencies (UZoJ), in the scale, incremental updates, and concurrent retrieval of the deployed service and the clarity of its APIs (CFfg, UZoJ), in the alignment of our benchmarks with LKM's retrieval and QA functions (CFfg), and in the matched-paper graph ablation and the separate evaluation of evidence retrieval, citation quality, and answer accuracy (7mr2).

The reviews led us to run several new experiments and to state more precisely what LKM does well and where it falls short. Details are in the individual responses.

  1. Evidence discovery on SciFact-Open (7mr2 W1, CFfg Q1). We split the original gold by how SciFact-Open collected it. The released retriever recovers more of the 460 original pairs (438 versus 156), largely because all 251 pooled pairs were annotated among its own top 50. On retriever-independent citation pairs the gap is smaller, and when candidates come from the benchmark corpus, LKM reranking recovers 367 original pairs and performs comparably on the citation pairs it indexes (90.8% versus 89.2%). LKM's gain comes from 662 additional model-consensus pairs, and an expert audit retained 194 of 200 sampled added labels. Section 5.1 now reports this breakdown, with the full analysis in a new appendix.

  2. Extraction fidelity (7mr2 W2, UZoJ Q3). We expanded the audit from 100 to 300 papers. Four domain experts checked every extracted conclusion and reasoning chain in 200 new papers against the full text. Errors that change meaning affected 2.0% of conclusions and 5.6% of reasoning chains, or 1.35% of all 11,993 extracted objects. We added the results to Section 3 and the full protocol to a new appendix.

  3. What reasoning graphs contribute (7mr2 Q2, UZoJ W2). In a token-matched control on all 208 ScholarQABench questions, compact graphs exceed source passages in answer quality on ScholarQA-Multi with 40% of the tokens, although passages score higher when citations are checked against the same OCR text. With two further writers, Gemini 2.5 Flash and DeepSeek-V4-Flash, graphs raise citation F1 by 2.0 to 3.8 points on both subsets, so the effect does not depend on GPT-4o.

  4. Cross-model QA and agent synthesis (UZoJ W2, CFfg W2). Replicating QA with three further answering models gives smaller, model-dependent differences. Following CFfg's suggestion, we evaluated a fixed agent on SciConBench over two independent sets of 100 questions: with LKM it covers more reference facts than with commercial paper search or web search, but does not consistently improve factual F1. We report both results with these caveats in Section 5.3 and the appendix.

  5. How the landscapes are used (CFfg W1, UZoJ Q1). We redrew Figure 3 to show how extraction, cross-paper organization, and search feed concrete applications, and added these applications to Section 4.2: workflow families have supplied about 17,000 candidate verifiable tasks, question families helped researchers formulate about 50 open or semi-open research problems, and the Evidence Engine serves a deployed manuscript-review agent. Section 5 now states that the benchmarks use paper-level retrieval and reasoning graphs.

  6. Incremental updates and system details (UZoJ W1, 7mr2 Q1 and Q3, CFfg W3). In production, the median time from OCR completion to extraction completion is 85 minutes, graph insertion follows within a median of 45 seconds, and extraction costs about CNY 0.55 per paper. We also describe the source corpus (Section 4.1), the input boundary for figures, tables, and equations (Section 3.2), and how typed edges trace a disputed claim to its source and dependent conclusions (Section 3).

  7. Consolidated protocol (UZoJ Q2). A new appendix gives the models, evidence budgets, metrics, and judges for every benchmark, the rationale for each model choice, and per-model replication results.

We are grateful for the reviewers' time and expertise. Their comments have substantially improved the paper, and we would be glad to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.