- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ParaKG is a parallel pipeline for constructing a corpus-level knowledge graph from scientific documents. It distributes entity and relation extraction across LLM workers, performs local deduplication, and combines document graphs through hierarchical merging with a shared alias cache. The experiments use two materials-science corpora and examine chunk size, concurrency, model choice, extraction and merging time, and changes in graph size as parallelism increases.
Critical strengths
The paper addresses a practical systems problem and provides measurements for both extraction and corpus-level integration. On BATTERY-173, increasing the number of agents from 1 to 16 reduces total runtime from 26.05 to 3.34 hours (Figure 5). The larger run processes 3,505 documents in approximately 12.5 hours with 256 agents. The breakdown by stage is useful: it shows how merging becomes a bottleneck and where additional parallel resources stop producing proportional gains. The discussion of execution-order effects in the shared cache is also candid and relevant to anyone implementing this type of pipeline.
Critical weaknesses
My main concern is that the evaluation does not establish whether the faster pipeline preserves graph correctness. Node counts, graph density, and deduplication rates cannot distinguish successful alias resolution from incorrectly merging distinct entities. They also cannot establish whether relations have been extracted correctly or important information has been lost. This matters particularly because concurrency changes the resulting graph and the paper makes claims about preserving precision. Sections 4.3 and 4.5 acknowledge that extraction and merge accuracy remain unevaluated, but these are central to assessing the contribution. A focused manual evaluation of extracted facts and merge decisions would already be valuable. The experiments also leave the benefit of the proposed architecture insufficiently isolated. The serial version is a useful reference for measuring scaling, but the evaluation does not compare against a simpler parallel implementation using the same GPU allocation. There is also no targeted ablation of the hierarchical merger and shared cache. Consequently, the results show that the implementation benefits from parallel resources, while the additional value of these architectural choices remains uncertain.
Questions for authors
- Can you provide a manually checked evaluation of extracted relations and entity merges, including a comparison between serial and parallel configurations? In particular, how often does deduplication merge genuinely equivalent entities versus collapse distinct entities?
- Using the same extracted local graphs and GPU allocation, how does the proposed merger compare with a simpler parallel merge strategy? Wall-clock time and the number of LLM verification calls would help establish the contribution of the hierarchy and cache.
Hi reviewers, please be responsive
Dear reviewers,
The authors have provided detailed rebuttal. Please respond to authors' rebuttal, If you have not done so. Thanks!
AC