acceptodds
Under review as a conference paper at ICLR 2027

ParaKG: Scalable Corpus-Level Knowledge Graph Construction from Scientific Documents with Parallel LLM Agents

Abstract

Knowledge graphs (KGs) are essential for organizing large collections of unstructured scientific literature. While existing LLM-based methods can effectively construct local KGs from individual documents, scaling to corpus-level graphs poses both severe computational bottlenecks and cross-document integration challenges. To address these challenges, we propose **ParaKG**, a parallel multi-agent framework for large-scale scientific KG construction. ParaKG distributes the document extraction workload across dynamically scheduled agents and merges the resulting local graphs via a hierarchical binary-tree reduction. To scale entity resolution without sacrificing precision, we introduce a two-tier mechanism combining fast lexical filtering with context-grounded LLM verification and persistent caching. We evaluate ParaKG on real-world materials science corpora, demonstrating linear speedups during document-level extraction. Our analysis also characterizes sub-linear scaling bottlenecks during the global merge phase, investigates non-deterministic concurrency effects on graph topology, and evaluates stage-specific performance trade-offs across diverse large language model architectures.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
knowledge graph, multi-agent system, large language model, parallel computing, distributed computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4cBnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ParaKG is a parallel pipeline for constructing a corpus-level knowledge graph from scientific documents. It distributes entity and relation extraction across LLM workers, performs local deduplication, and combines document graphs through hierarchical merging with a shared alias cache. The experiments use two materials-science corpora and examine chunk size, concurrency, model choice, extraction and merging time, and changes in graph size as parallelism increases.

Critical strengths

The paper addresses a practical systems problem and provides measurements for both extraction and corpus-level integration. On BATTERY-173, increasing the number of agents from 1 to 16 reduces total runtime from 26.05 to 3.34 hours (Figure 5). The larger run processes 3,505 documents in approximately 12.5 hours with 256 agents. The breakdown by stage is useful: it shows how merging becomes a bottleneck and where additional parallel resources stop producing proportional gains. The discussion of execution-order effects in the shared cache is also candid and relevant to anyone implementing this type of pipeline.

Critical weaknesses

My main concern is that the evaluation does not establish whether the faster pipeline preserves graph correctness. Node counts, graph density, and deduplication rates cannot distinguish successful alias resolution from incorrectly merging distinct entities. They also cannot establish whether relations have been extracted correctly or important information has been lost. This matters particularly because concurrency changes the resulting graph and the paper makes claims about preserving precision. Sections 4.3 and 4.5 acknowledge that extraction and merge accuracy remain unevaluated, but these are central to assessing the contribution. A focused manual evaluation of extracted facts and merge decisions would already be valuable. The experiments also leave the benefit of the proposed architecture insufficiently isolated. The serial version is a useful reference for measuring scaling, but the evaluation does not compare against a simpler parallel implementation using the same GPU allocation. There is also no targeted ablation of the hierarchical merger and shared cache. Consequently, the results show that the implementation benefits from parallel resources, while the additional value of these architectural choices remains uncertain.

Questions for authors
  1. Can you provide a manually checked evaluation of extracted relations and entity merges, including a comparison between serial and parallel configurations? In particular, how often does deduplication merge genuinely equivalent entities versus collapse distinct entities?
  2. Using the same extracted local graphs and GPU allocation, how does the proposed merger compare with a simpler parallel merge strategy? Wall-clock time and the number of LLM verification calls would help establish the contribution of the hierarchy and cache.
Reviewer_tdPwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies parallel corpus-level knowledge graph construction from scientific documents and proposes ParaKG. It combines two-level extraction scheduling, hierarchical binary merging, and entity resolution using lexical filtering, LLM verification, and an alias cache. Experiments on corpora of 173 and 3,505 materials-science papers report stage-specific runtimes, deduplication scale, and graph-structure variation across concurrency settings.

Critical strengths

A concrete corpus-integration problem. The work addresses both large-volume document extraction and cross-document alias integration, two central stages of scientific KG construction. Operational-scale runs and structural variation are reported. The paper provides an end-to-end run on 3,505 documents and explicitly reports that final node counts are not identical across concurrency configurations.

Critical weaknesses

The precision-preservation claim lacks direct quality measurements. The authors acknowledge the absence of a gold standard in Sections 4.2 and 4.5; this is a disclosed limitation, not an overlooked validation. Nevertheless, node counts and deduplication rates do not test the abstract’s claim of scaling entity resolution without sacrificing precision. Maybe assess human-verified entity matches and triples, including false merges, missed matches, and extraction errors, can establish the quality retained by the faster pipeline.

Component-level performance attribution is limited. The fixed-workload curves across increasing GPU counts are valid evidence of parallel scaling, and the request-limit sweep also studies scheduling behavior. They do not separately quantify the added benefits of the proposed scheduler, binary reduction, and alias cache. A simpler scheduling or merging reference at the same hardware allocation, with comparable output quality, would strengthen attribution to these components without discounting the demonstrated scaling results.

Canonicalization and reconciliation rules need a more precise specification. Section 3.3 explains that LLM alignment selects canonical representatives and the main orchestrator reconciles the shared alias map; Section 4.5 explicitly acknowledges order dependence. What remains unspecified is how incompatible mappings proposed by different merges would be reconciled and whether previously remapped vertices and edges are updated. Please specify these rules and assess changes in equivalence decisions and edge connectivity across repeated runs. Node-count variation alone does not demonstrate incorrect merging.

Questions for authors

see as above.

Reviewer_3Hv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to build one knowledge graph from a large collection of scientific papers faster, by running many LLM agents in parallel. The authors propose ParaKG, which splits each paper into chunks, extracts entities and relations from the chunks in parallel, and then merges the per-paper graphs pair by pair in a binary tree. For merging, a cheap lexical filter groups possible duplicate entities and an LLM decides which ones are the same, with a cache to skip repeated decisions. They test it on two materials science corpora of different sizes. Extraction speeds up well as more agents are added, but merging scales much worse and becomes the main bottleneck. They also compare a few open-weight LLMs on runtime, and show that the merged graph comes out slightly different depending on the number of agents. The paper does not evaluate the quality of the resulting graphs, which the authors leave for future work.

Critical strengths
  1. The split of runtime into extraction and merging is useful. It shows that extraction speeds up almost linearly up to 8 agents, while merging slows things down as the tree gets closer to the root. Anyone building a similar pipeline would want to know this.

  2. The system was run on a fairly large corpus. They built a graph from 3,505 papers with 256 agents in about 12.5 hours, going from 1.41M raw nodes down to 416.5K after merging.

  3. The parameter sweeps are practical. For example, going above 8 requests per agent hardly changes anything because the 16 paper workers can't send tasks fast enough, and the reasoning model takes about 13 hours to merge while Gemma 4 takes about 2.

  4. The authors are upfront about the limits. They say there is no gold-standard evaluation, that merging scales sub-linearly and that the merged graph changes a bit with the number of agents. The prompts are included and only open-weight models are used, so others can try the same setup.

Critical weaknesses
  1. The quality of the graph is never measured. Nothing checks whether the extracted triples or the merges are correct, and the authors admit this, yet the conclusion says the system 'resolves aliases effectively'. Merging removes about 32% of nodes on BATTERY-173 but over 70% on XAS-3.5K, so without some evaluation it's impossible to tell good deduplication from over-merging.

  2. There are no real baselines. The only baseline is the same system with one agent. There is nothing against existing pipelines like GraphRAG indexing, KGGen, EDC or AutoSchemaKG, and nothing against a simpler merge, like clustering all entities once and resolving the clusters in parallel, which could avoid the bottleneck at the top of the tree.

  3. The novelty is limited. Parallel calls to vLLM servers are normal batching, and the two-tier merge is the usual blocking plus matching idea from entity resolution. The 'agents' don't plan, use tools or talk to each other. AutoSchemaKG, which the paper cites, already builds corpus-level graphs from over 50 million documents.

  4. The scaling claims are a bit strong. The abstract says extraction scales linearly, but that only holds up to 8 agents. The full 16-agent run is about 7.8 times faster than one agent while using roughly twice the GPU hours.

  5. The 256-agent run isn't a clean scaling test. It is compared with BATTERY-173, which is a different corpus in a different format (JSONL vs PDF), and there are no smaller runs on XAS-3.5K. The numbers that are there suggest weak per-agent scaling, with 16 times the agents giving about 2.5 times the throughput (5,498 vs 2,238 chars/s), so the claim that it 'scales effectively' isn't really tested.

  6. The variation across agent counts can't be pinned on the cache. The paper blames the roughly 600-node difference between 4 and 8 agents on the non-deterministic cache, but the extracted counts also change between agent counts (119.6K at 16 agents vs 119.7K elsewhere), and as the paper itself notes, equal counts do not guarantee identical node sets, so the inputs to the merge may differ even at 4 and 8 agents. There are no repeated runs or error bars. It's also unclear how this fits with the earlier description of the cache as deterministically reconciled. Separately, Table 2's 16-agent node counts (119.7K and 81.7K) don't match Figure 6 (119.6K and 82.3K).

Questions for authors

Questions

  1. Are 'don't merge' decisions cached? From the merge algorithm, only successful merges are saved, so a cluster with two genuinely different entities, like a material and its doped version, seems to miss the cache every time. What is the cache hit rate, and how many LLM calls does the merge stage make in total?

  2. What happens when merges chain? Since the cache doesn't follow chains, if one merge maps A to B and a later one maps B to C, could A and B end up as separate entities after the final merge? And how are conflicting mappings, like A to B and A to C, resolved?

  3. Figure 6 suggests the extraction was rerun for each agent count. Can you confirm this, and were the extracted node sets at 4 and 8 agents actually identical, or only the counts?

  4. Apart from the context length and chunk size, I couldn't find the exact model checkpoints, the inference settings (temperature, output length, output format) or the clustering thresholds. Could you give these?

Experiments (Experiments 5 and 6 can strengthen the work materially)

  1. A quality check of the graph. A few hundred sampled merges, non-merges and triples per corpus, checked by hand or by a strong LLM judge, would show both over-merging and missed merges, and whether the 70% node reduction on XAS-3.5K is correct.

  2. A comparison with a simpler merge and one existing system. Clustering all entities once and resolving the clusters in parallel, plus a run of something like KGGen or EDC on the same corpus with runtime and quality reported, would show whether the binary tree, and the system as a whole, actually help.

  3. A same-corpus scaling point. Even one smaller run on XAS-3.5K, for example with 16 agents, with GPU hours reported, would give a fair comparison for the 256-agent result.

If you address all the above questions and experiments, I may reconsider my score.

Comments on the paper

1 comment
Area_Chair_pLqjOpenReview ↗

Hi reviewers, please be responsive

Dear reviewers,

The authors have provided detailed rebuttal. Please respond to authors' rebuttal, If you have not done so. Thanks!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.