acceptodds
Under review as a conference paper at ICLR 2027

HAVE WE SEEN THIS BEFORE? EVALUATING AND IMPROVING INCREMENTAL KNOWLEDGE GRAPH CONSTRUCTION

Abstract

Knowledge graphs organize information from multiple sources for querying and reasoning. As new text arrives, maintaining a coherent graph requires integrating equivalent information without duplicating it and preserving distinct information without incorrectly merging it. Evaluating these complementary requirements is challenging because the appropriate update depends on what the graph already contains. We introduce DéjàKG, a controlled benchmark spanning initial Construction, Consolidation of equivalent information, and Extension with distinct new information. Single-fact inputs and predefined identity relationships specify expected graph updates, enabling deterministic, case-wise scoring without an additional large language model (LLM) judge. To address these integration challenges, we introduce IDEM, a framework that determines whether incoming entities and facts match existing knowledge or should be added to the graph. It compares their surface forms and descriptions using vector similarity, LLM-based decisions, or a hybrid of the two.

Primary area:
datasets and benchmarks
Keywords:
incremental knowledge graph construction, entity resolution, fact deduplication, benchmarks, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fi1tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is poorly written and difficult to follow. Its fragmented organization, excessive protocol-specific terminology, and template-like exposition make it read more like the output of an automated LLM-based research pipeline than a carefully developed scientific paper.

It studies incremental knowledge graph construction, where a system must decide whether entities and facts extracted from new text should reuse existing graph elements or be added as new information. the paper introduces DéjàKG, a controlled benchmark that separates this process into three tasks. It contains 7,500 single-fact inputs across development and test splits and evaluates systems by comparing the expected and observed numbers of newly added entities and facts.

Critical strengths

No specific strengths.

Critical weaknesses

The paper is poorly written and difficult to follow. Its fragmented organization, excessive protocol-specific terminology, and template-like exposition make it read more like the output of an automated LLM-based research pipeline than a carefully developed scientific paper.

Questions for authors

Can the authors compare IDEM with existing entity-resolution and incremental text-to-KG methods?

Can the benchmark be expanded beyond controlled single-fact inputs?

2 comments

Dear Reviewer fi1t,

Thank you for taking the time to review our paper. We have uploaded a revised manuscript with clearer benchmark and method descriptions, simplified terminology, and fewer repeated implementation details. Below, we also report additional Test results.

Regarding comparisons with existing methods, we tuned iText2KG on Dev using mean VU over three runs and fixed the selected settings before Test evaluation. In that Dev study, tuning improved its mean VU from 40.21% under the implementation defaults to 57.63%.

We have also completed two additional independent IDEM Test extractions. The updated results cover three separate extraction realizations, with extraction outputs shared across IDEM policies within each run. Both systems use the same pre-integration single-triple check.

The updated mean Test VU is 51.73% for iText2KG, 61.24% for IDEM vector, 91.52% for IDEM LLM-based, and 90.64% for IDEM hybrid with both automatic steps enabled. We will incorporate these results and their sample standard deviations into Section 6.2 in the next manuscript revision. Appendix B.3 documents the Dev tuning procedure.

We are evaluating KGGen as an additional comparison, applying its native aggregation and full-graph deduplication sequentially. We will update the manuscript with its results once they are available.

Regarding expansion beyond single-fact inputs, we would first like to explain why we chose this input format. We initially considered providing triples to evaluate how systems integrate them into an existing KG. However, systems such as iText2KG and IDEM generate labels or descriptions from source text and use them during integration. Triple-only inputs would leave out this source-dependent processing. We therefore provide text expressing one intended fact, allowing systems to generate these attributes while keeping the information to be integrated controlled.

The restriction applies to the incoming text. Each Test update is evaluated against the system’s actual graph built from 1,000 Construction inputs, without supplying gold anchor or identity labels.

With multi-fact passages, systems may extract different sets of facts before integration begins. To interpret their subsequent updates, we would need to track both what was extracted and how each claim was integrated. We are considering an extension in this direction, including cross-sentence coreference.

We have clarified this rationale in Section 3.1 and the scope in Section 6.5. The current benchmark is intended to complement extraction evaluations such as CaRB and Text2KGBench; its results do not establish performance on complex documents.

Thank you again for your feedback. We would be happy to clarify any parts of the revised manuscript that remain difficult to follow.

Thanks for your responses, I will maintain my original score.

Reviewer_wGXPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses entity and relation resolution during knowledge graph construction. It introduces a framework (IDEM) that resolves entities and relations and a new benchmark (DéjàKG) to measure resolution quality. IDEM is then evaluated against DéjàKG. Of IDEM's three modes (embedding-based, LLM-based, hybrid), the two modes involving an LLM judge outperform the purely embedding-based method.

DéjàKG works by presenting sentences that describe facts involving a subject, an object and a relation, and asking the framework under test how many new entities and relations it would create for this fact. Asking for a pure entity count allows arbitrary graph representation and avoids an LLM judge, but it may miss errors.

After a construction phase with purely disjoint facts, we freeze the graph and evaluate the update steps in parallel on it. The dataset is constructed to test several difficult resolution scenarios, such as antonyms, synonyms, homonyms, and sibling entities that share the same relation to another entity. We collect three scores: success during graph construction, success during updates that should reuse entities and relations, and success during updates that should reuse nothing. These three scores are then combined via harmonic mean into the VU score (Valid Update).

IDEM integrates text-extracted data into a knowledge graph. It works by first extracting triples via LLMs, then generating descriptions for entities and relations from the source and the triples, and finally one of three resolution processes:

  • The vector policy embeds the entities’ names and descriptions separately via an embedding model, then normalises, scales and concatenates the two vectors into a higher-dimensional unit vector (doubling the dimension), which then matches an existing entity if their cosine similarity exceeds a threshold.
  • The LLM-based policy uses the same embedding but retrieves the top-k most similar vectors and lets the LLM judge whether one matches the new entity.
  • The identity-hybrid policy follows the LLM-based approach until the top matching entity exceeds an even higher threshold and has a sufficient margin over the runner-up.

The three methods are evaluated against DéjàKG, as well as an iText2KG-based implementation (Appendix C.2), with the LLM-based and hybrid approaches performing best.

Critical strengths
  • The benchmark DéjàKG seems carefully constructed and well overseen, both by humans and AI.
  • The dataset covers multiple different cases in which entity resolution might go wrong (Acronym, Alias, Inverse, Multilingual, Relation synonym, Homonym, Antonym, Near entity, Sibling, Different value).
  • This is valuable for any KG construction system to identify which areas need improvement.
Critical weaknesses
  • I'm unsure how good IDEM’s VU scores should be considered. The only comparison is iText2KG (in an appendix), and it's against the 6000 development data points used to tune IDEM’s parameters, not the 1500 held-out data points. The failure examples make it clear that iText2KG's resolution thresholds were set too low; it confused Germany and Austria, and India and Pakistan.
  • iText2KG was also not built with this test in mind; it already often failed in the construction phase, where it did not always extract exactly one triple per sentence.
  • An interesting point of comparison would be a “random guesser” that picks one of the 4 possible answers at random (2 new entities, 1 new relation; 1-1; 0-1; 0-0), using the development set to fine-tune the probabilities.
  • Another point of comparison could have been the impact of the model in use. gpt-5.6-luna is cutting-edge among cheap models, while text-embedding-3-large is two and a half years old, which might already explain part of the gap between the vector policy and the LLM-based policies.
  • LLMs have just been optimised more than embeddings in recent years. Using a similarly old LLM might give interesting insights into whether the procedure or the model makes the difference.
Questions for authors

DéjàKG restricts inputs to single-fact (subject–predicate–object) sentences; i.e., single-fact probes do not evaluate cross-sentence coreference, interacting claims, or context-dependent interpretations, which are realistic for document-level integration. Q1: To what extent does performance on single-fact inputs correlate with the ability to handle multi-fact documents or paragraphs with complex co-references?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wGXP: Updated Comparisons and Evaluation Clarifications

Dear Reviewer wGXP,

Thank you for your careful review and constructive suggestions. We appreciate your recognition of the benchmark's diagnostic value and category coverage. Your comments prompted us to strengthen the baseline comparison and clarify what the evaluation measures. We have uploaded a revised manuscript. Below, we also report additional Test results.

  1. Baseline tuning and held-out comparison

We agree that the original development-only comparison was insufficient. We tuned iText2KG on Dev using three-run mean VU, then fixed the selected settings for Test.

The selected entity-name/entity-label weights are 0.80/0.20 and entity/relation thresholds are 0.70/0.60. In that Dev study, mean VU improved from 40.21% under implementation defaults to 57.63%.

We have also completed two additional independent IDEM Test extractions. The updated results cover three separate extraction realizations, with outputs shared across IDEM policies within each run. Both systems use the same pre-integration single-triple check.

Updated Test VU (mean ± sample SD, %) is 51.73 ± 0.20 for iText2KG, 61.24 ± 0.12 for IDEM vector, 91.52 ± 1.14 for IDEM LLM-based, and 90.64 ± 0.91 for IDEM hybrid with both automatic steps enabled. These results will be incorporated into Section 6.2 in the next revision. Appendix B.3 documents Dev tuning.

KGGen evaluation is also ongoing.

  1. Single-fact inputs, fairness, and document-level generalization

We appreciate your concern that iText2KG was not designed to produce exactly one triple. We retain its own extraction procedure. Under our shared output-granularity requirement, legitimate additional facts can nevertheless cause a case to fail. We now state this limitation explicitly: the comparison measures graph-update performance under shared granularity, rather than unrestricted document extraction.

Our aim is to examine how incoming information is integrated into a system-built KG. Each Test update is evaluated against the graph built from 1,000 Construction inputs, without supplying gold anchor or identity labels. Extension can reuse existing nodes while adding a distinct fact. Text allows systems to generate contextual labels or descriptions for integration. Controlling each input to one intended claim makes the expected update explicit and reduces confounding from omitted claims, additional facts, and errors across claims that offset one another in aggregate counts.

In direct answer to your question, we have not measured the correlation between single-fact scores and performance on multi-fact documents with complex coreference. We therefore do not claim such transfer. We have clarified that DéjàKG complements extraction benchmarks such as CaRB and Text2KGBench, with document-level extension requiring additional annotations and claim-level observations.

  1. Interpretation of a random-guesser baseline

Thank you for this suggestion. Our benchmark evaluates actual graph updates: systems select existing entities and facts to reuse or create new ones, and we measure the resulting additions. Reuse therefore requires selecting specific nodes and facts.

A random graph-building baseline could select a retrieved candidate or NEW at each step. Correct reuse first requires the correct node to appear among the candidates. With exactly k candidates plus NEW, uniform random selection then chooses that node with probability 1/(k+1), conditional on its retrieval. For k=3, this is 1/4 per endpoint. Overall success also depends on retrieval coverage, correct choices for both endpoints, and the subsequent fact decision.

Using IDEM's candidates would retain its descriptions, embeddings, and retrieval weights. Such a control would test random selection after IDEM's retrieval. Guessing one of four increment pairs directly would instead test the count distribution, without performing graph construction.

These identity-selection probabilities are not chance levels for VU: incorrect selections can still produce the expected counts. We are undertaking a human audit to examine this gap. No random-control result is included yet.

  1. LLM and embedding model effects

Thank you especially for highlighting this potential confound, which we had not adequately examined. We agree that the observed gap may reflect both the matching policy and the capabilities of the underlying models.

We used text-embedding-3-large (https://developers.openai.com/api/docs/models/text-embedding-3-large), documented as OpenAI's most capable embedding model. We are designing a comparison with an older LLM, replacing the LLM throughout extraction, description generation, and integration while retaining the embedding model and remaining settings. We will update the paper with the results once the comparison is complete.

Thank you again for these helpful suggestions. We welcome further questions and will update the discussion as the ongoing analyses become available.

Reviewer_fojwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a core, under-studied problem in incremental knowledge graph construction: deciding whether incoming information should be reused (to avoid duplication) or added (to avoid incorrect merging), based on what the graph already contains.

Critical strengths

The paper is well-structured and clearly written : the three-task decomposition and the 10-category design are introduced with concrete, intuitive examples (Table 3), making the benchmark's intent immediately graspable.

By reducing each single-fact update to a check of expected entity/fact increments, the protocol avoids the variability and cost of LLM-as-judge while remaining robust to differences in how systems express the same information. The independent-probe design cleanly isolates single-update behavior from cumulative stream effects.

Critical weaknesses

The authors demonstrate multiple cases where counts match but content is wrong (reversed relation direction, antonymous relations falsely merged). Without even a small-scale human-annotated semantic validation, the VU score's validity as a measure of incremental construction quality remains uncertain.

Only iText2KG is evaluated, and solely on the development set. The absence of KGGen, CESI, EDC, and other canonicalization/resolution methods prevents readers from assessing IDEM's standing relative to the state of the art and limits the benchmark's value as a comparative tool.

Single-fact inputs exclude cross-sentence coreference, interacting claims, and context-dependent interpretation — all central to real-world KG construction. While the paper frames this as a deliberate boundary, it substantially restricts what can be inferred about system behavior in realistic settings.

Questions for authors
  1. Could you provide a human-annotated semantic correctness analysis on a subset of count-correct updates (e.g., 200–500 cases) to quantify the proportion for which matching counts correspond to semantically correct reuse/addition? This would substantially strengthen the validity claims of the VU metric.
  2. What is the availability and identity of "gpt-5.6-luna" ? Is it a publicly accessible model or an internal/preview API? The reproducibility statement notes that exact reproduction is not guaranteed; could you clarify whether external researchers can access the same model configuration?
  3. Why is the baseline comparison limited to iText2KG on the development set only ? Could you add at least one or two representative methods (e.g., KGGen or EDC) on the test set so that IDEM's effectiveness can be assessed against the broader state of the art?
1 comment
AuthorsOpenReview ↗

Response to Reviewer fojw: Clarifications, Updated Results, and Ongoing Validation

Dear Reviewer fojw,

Thank you for taking the time to read our paper and provide thoughtful feedback. We appreciate your recognition of the integration problem and the controlled design. We have uploaded a revised manuscript. Below, we also report additional Test results and outline the work still underway.

  1. Semantic validation of the count-based evaluation protocol

We agree that count agreement needs semantic validation. An update may reuse an incorrect entity or merge distinct claims while producing the expected increments. Your suggested human analysis would quantify this gap and help interpret VU.

We are undertaking this analysis using stored IDEM outputs as observations. We sample count-correct updates across categories and inspect the input, extracted triple, relevant Construction information, and actual entities and facts reused or added. We will report the proportion that are semantically correct, together with error types and representative cases, identifying the policies and samples covered.

Our procedure uses an initial human judgment, a subsequent LLM review, and final human adjudication of disagreements. We will disclose the author's annotator role, LLM assistance, sampling procedure, coverage, and unresolved cases. The analysis is ongoing; its quantitative results and examples will be added to the appendix when complete.

  1. Model identity and accessibility

Thank you for pointing out the need to clarify this. gpt-5.6-luna is an OpenAI API model with official documentation (https://developers.openai.com/api/docs/models/gpt-5.6-luna). We have explicitly identified OpenAI and added this link to the main settings table.

The reported configuration uses low reasoning effort and text-embedding-3-large embeddings with 3,072 dimensions. The reproducibility caveat refers to LLM response variability: even with the same model, settings, instructions, and inputs, repeated calls can produce different outputs, graph updates, and scores. Count scoring is deterministic for fixed outputs. The updated Test results below report means and sample standard deviations across three separate extraction realizations.

  1. Development-tuned, held-out baseline comparison

We have completed Dev tuning and held-out Test evaluation of iText2KG, with settings fixed before Test. We also completed two additional independent IDEM Test extractions, bringing the comparison to three separate extraction realizations. Both systems use the same pre-integration single-triple check.

Updated Test VU (mean ± sample SD, %) is 51.73 ± 0.20 for iText2KG, 61.24 ± 0.12 for IDEM vector, 91.52 ± 1.14 for IDEM LLM-based, and 90.64 ± 0.91 for IDEM hybrid. We will incorporate these results into Section 6.2 in the next revision. Dev tuning is documented in Appendix B.3.

KGGen evaluation is ongoing. We will update the manuscript with its results once they are available.

  1. Single-fact design and document-level generalization

We have clarified the input design in Section 3.1 and document-level limitations in Section 6.5.

DéjàKG focuses on integrating incoming information into a system-built KG. Each Test update is evaluated against the graph built from 1,000 Construction inputs, without supplying gold anchor or identity labels. We retain text because systems such as iText2KG and IDEM derive contextual labels or descriptions used in integration; reference triples alone would bypass those steps.

We control each input to one intended claim to make its expected update explicit. With multiple claims, differences in what is extracted, including omitted claims and legitimate additional facts, can confound the interpretation of integration outcomes. Errors on different claims can also offset one another in aggregate counts. Single-fact inputs reduce these ambiguities, while retaining extraction and contextual representation generation within the evaluation.

DéjàKG therefore complements extraction benchmarks such as CaRB and Text2KGBench. Its results do not establish performance on interacting claims or cross-sentence coreference. Extending the evaluation to multi-claim inputs with claim-level observations and semantic validation is an important direction we are exploring.

Thank you again for these helpful suggestions. We welcome further questions and will follow up with the ongoing validation results when available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.