- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper is poorly written and difficult to follow. Its fragmented organization, excessive protocol-specific terminology, and template-like exposition make it read more like the output of an automated LLM-based research pipeline than a carefully developed scientific paper.
It studies incremental knowledge graph construction, where a system must decide whether entities and facts extracted from new text should reuse existing graph elements or be added as new information. the paper introduces DéjàKG, a controlled benchmark that separates this process into three tasks. It contains 7,500 single-fact inputs across development and test splits and evaluates systems by comparing the expected and observed numbers of newly added entities and facts.
Critical strengths
No specific strengths.
Critical weaknesses
The paper is poorly written and difficult to follow. Its fragmented organization, excessive protocol-specific terminology, and template-like exposition make it read more like the output of an automated LLM-based research pipeline than a carefully developed scientific paper.
Questions for authors
Can the authors compare IDEM with existing entity-resolution and incremental text-to-KG methods?
Can the benchmark be expanded beyond controlled single-fact inputs?
2 comments
Thanks for your responses, I will maintain my original score.
Dear Reviewer fi1t,
Thank you for taking the time to review our paper. We have uploaded a revised manuscript with clearer benchmark and method descriptions, simplified terminology, and fewer repeated implementation details. Below, we also report additional Test results.
Regarding comparisons with existing methods, we tuned iText2KG on Dev using mean VU over three runs and fixed the selected settings before Test evaluation. In that Dev study, tuning improved its mean VU from 40.21% under the implementation defaults to 57.63%.
We have also completed two additional independent IDEM Test extractions. The updated results cover three separate extraction realizations, with extraction outputs shared across IDEM policies within each run. Both systems use the same pre-integration single-triple check.
The updated mean Test VU is 51.73% for iText2KG, 61.24% for IDEM vector, 91.52% for IDEM LLM-based, and 90.64% for IDEM hybrid with both automatic steps enabled. We will incorporate these results and their sample standard deviations into Section 6.2 in the next manuscript revision. Appendix B.3 documents the Dev tuning procedure.
We are evaluating KGGen as an additional comparison, applying its native aggregation and full-graph deduplication sequentially. We will update the manuscript with its results once they are available.
Regarding expansion beyond single-fact inputs, we would first like to explain why we chose this input format. We initially considered providing triples to evaluate how systems integrate them into an existing KG. However, systems such as iText2KG and IDEM generate labels or descriptions from source text and use them during integration. Triple-only inputs would leave out this source-dependent processing. We therefore provide text expressing one intended fact, allowing systems to generate these attributes while keeping the information to be integrated controlled.
The restriction applies to the incoming text. Each Test update is evaluated against the system’s actual graph built from 1,000 Construction inputs, without supplying gold anchor or identity labels.
With multi-fact passages, systems may extract different sets of facts before integration begins. To interpret their subsequent updates, we would need to track both what was extracted and how each claim was integrated. We are considering an extension in this direction, including cross-sentence coreference.
We have clarified this rationale in Section 3.1 and the scope in Section 6.5. The current benchmark is intended to complement extraction evaluations such as CaRB and Text2KGBench; its results do not establish performance on complex documents.
Thank you again for your feedback. We would be happy to clarify any parts of the revised manuscript that remain difficult to follow.