acceptodds
Under review as a conference paper at ICLR 2027

Yesterday's Facts, Today's Violations: Schema Tightening in LLM-Written Knowledge Graphs

Abstract

LLM agents increasingly write into the knowledge graphs that later ground their answers, and validating each write against a schema is already production practice. In every such system the schema is monotonic: it is frozen or only extended, so data that once passed validation keeps passing. The constraints users ask for run the other way. Writing for the set of graphs that validate against shapes , “every contract must carry at least one document” is a tightening, a change with . It retroactively invalidates data that has not changed, along with answers already delivered from that data. Incremental SHACL validation does not cover this case, because it assumes that the data changes while the shapes stay fixed. We treat a constraint change as a first-class, versioned event. After a policy gate and an impact preview, the system revalidates the whole graph and emits each newly broken fact as a conflict event that records the shape version that judged it and the agent behavior that wrote the value; the value is then repaired, or the event is deferred with the violated shape as its reason. No existing benchmark specifies what breaks under a tightening or what the correct repair is, so we build one by inversion: we delete known values, inject tightenings, and diff the validation reports, and each deleted value becomes the gold repair. On QUDT and LUBM, a single tightening takes the report from to violations without touching the data. Our full loop clears of the violations it attempts but repairs only correctly, and the gap between the two widens from to  pp as repair prompts get stronger. A system that reports validity alone therefore overstates its repair rate and silently corrupts the graph it is meant to protect.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
knowledge graphs, retrieval-augmented generation, LLM agents, neuro-symbolic, non-monotonic reasoning, hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hWvfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies schema tightening in LLM-written knowledge graphs, proposing a versioned validation-and-repair workflow and a deletion-based benchmark. It distinguishes constraint validity from restoration accuracy and investigates constraint-assisted question answering.

Critical strengths
  1. The paper addresses a meaningful problem in knowledge graph maintenance.
  2. The distinction between valid and correct repairs is valuable, and the authors report informative negative results.
Critical weaknesses
  1. The experiments validate only part of the proposed full loop. Appendix D.6 explicitly leaves approval and producer tracing unevaluated, and Appendix E.2 reports no human-resolution study. These are central workflow components, not merely additional evaluation settings. Their acknowledged omission leaves the added value of the complete workflow unestablished.
  2. The headline "widening gap" is not a paired measurement of valid-but-incorrect repairs. Full-loop validity uses 152 attempted tasks, whereas accuracy uses only 40 unique-gold attempts; attempted subsets also differ across methods. Disclosing these denominators does not make their difference a direct measure of repair errors. A common-task analysis with explicit abstention outcomes is needed.
Questions for authors
  1. Does the claimed gap still widen when validity and restoration are measured on exactly the same tasks, with abstentions accounted for?
  2. Which experimental result isolates the benefit of versioning or producer attribution beyond the validation-and-repair stages already tested?
Reviewer_YHRJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies schema tightening in LLM-written knowledge graphs, where changes to existing constraints can invalidate previously accepted facts. The authors propose a framework for approving schema changes, identifying newly introduced violations, routing them to their producers, and repairing or deferring them. They also construct a repair benchmark by deleting existing graph values and injecting new constraints. The experiments examine retroactive violations and repair performance on QUDT and LUBM, as well as the use of executable constraints for question answering. The results suggest that satisfying schema constraints does not necessarily mean recovering the correct facts. Executing rules also improves QA in some settings, particularly when retrieval is incomplete, although the gains are less consistent in the agentic retrieval experiments.

Critical strengths
  1. The problem is relevant and well motivated. The paper highlights a practical issue in maintaining LLM-written knowledge graphs: previously accepted data can become invalid when constraints change. Treating schema changes as explicit events is a useful perspective for persistent knowledge systems.
  2. The evaluation distinguishes constraint satisfaction from factual correctness. I find this distinction important, since a repair can satisfy the validator without restoring the intended fact. I also appreciate that the paper reports negative QA results and discusses when constraint execution provides little additional benefit.
Critical weaknesses
  1. The central repair results are difficult to interpret because validity and accuracy are evaluated on different populations. The paper emphasizes a widening gap between validity and accuracy, reporting 98.0% validity and 75.0% accuracy for the full method. However, Table 15 shows that these results have substantially different denominators: validity is measured over 152 attempted repairs, whereas accuracy is measured over only 40 attempted tasks with unique gold answers. Moreover, the full method defers 95 of the 244 repair tasks, and the attempted subsets differ across methods. All 70 tasks with unique gold answers also come from QUDT, leaving factual repair accuracy on LUBM unmeasured. Consequently, the reported 23-percentage-point gap does not directly represent the proportion of repairs that satisfy the constraints but restore incorrect facts. Similarly, changes in the gap across methods may partly reflect differences in the evaluated subsets rather than changes in repair quality. Since this distinction is central to the paper's empirical claims, a paired evaluation on a common task set, with deferrals and coverage reported separately, is needed to support the conclusions more convincingly.

  2. The experiments do not fully validate the proposed end-to-end workflow or establish the contribution of its individual components. Appendix D.6 explicitly states that the approval stage and producer tracing are designed but not evaluated. These are important parts of the proposed framework, particularly its claimed ability to attribute violations to responsible agent behaviors. The paper also acknowledges that identifying a focus node's named graph does not necessarily identify the behavior that wrote the offending value when multiple behaviors modify the same node. In addition, the deletion-based repair benchmark primarily measures restoration of original values under synthetically introduced violations. It remains unclear to what extent this setting captures the difficulties of real schema evolution, as opposed to conventional missing-value repair. Although the paper includes validation-feedback repair baselines, it does not isolate the incremental benefits of the approval, versioning, and producer-routing components beyond a straightforward SHACL revalidation and LLM-based repair pipeline. These gaps limit the empirical support for the proposed framework as an integrated system.

Questions for authors
  1. Could the authors report validity and accuracy on the same set of unique-gold tasks, including the number of valid but incorrect repairs and the number of deferrals? This would help establish whether the reported widening gap persists under a paired evaluation.
  2. Is there any evidence, even on a small multi-agent example, that the proposed producer attribution identifies the behavior actually responsible for a violation? This would help assess an important component of the claimed end-to-end framework.
  3. Could the authors clarify what the proposed framework offers beyond simply rerunning SHACL validation after a schema update and feeding the resulting violations to an existing LLM-based repair pipeline? A comparison with such a straightforward baseline would help establish the contribution of the proposed framework.
Reviewer_prnQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats schema tightening as a first-class, versioned event for LLM-written knowledge graphs. It proposes a write-path loop: propose/approve a patch, preview impact, revalidate the whole graph, emit each newly broken fact as a conflict event carrying the shape version and producer, then repair or defer. The evaluation uses QUDT and LUBM for retroactive repair and 2WikiMultiHopQA, MuSiQue, and BoardgameQA for QA.

Critical strengths

1.The paper correctly identifies that schema change in LLM-written KGs has been treated as extend-only, and that tightening is the case that can retroactively invalidate stored facts. 2.The result that validity rises to 98.0% while accuracy reaches only 75.0%—and that the gap widens as repair methods get stronger—is the paper’s most important finding.

Critical weaknesses

1.The write-path loop is not evaluated end-to-end. 2.Experiment is too thin for strong claims about repair accuracy. 3.The repair benchmark is synthetic and measures restoration, not correctness. 4.Read-path contribution is conditional and should be scoped more tightly. 5.False-alarm and priority handling remain unresolved.

Questions for authors

1.What is the accuracy of the full loop if deferrals are counted as failures, and what is the accuracy among automatically applied repairs only? 2.What is the false-alarm precision of the validator on BoardgameQA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.