- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies schema tightening in LLM-written knowledge graphs, proposing a versioned validation-and-repair workflow and a deletion-based benchmark. It distinguishes constraint validity from restoration accuracy and investigates constraint-assisted question answering.
Critical strengths
- The paper addresses a meaningful problem in knowledge graph maintenance.
- The distinction between valid and correct repairs is valuable, and the authors report informative negative results.
Critical weaknesses
- The experiments validate only part of the proposed full loop. Appendix D.6 explicitly leaves approval and producer tracing unevaluated, and Appendix E.2 reports no human-resolution study. These are central workflow components, not merely additional evaluation settings. Their acknowledged omission leaves the added value of the complete workflow unestablished.
- The headline "widening gap" is not a paired measurement of valid-but-incorrect repairs. Full-loop validity uses 152 attempted tasks, whereas accuracy uses only 40 unique-gold attempts; attempted subsets also differ across methods. Disclosing these denominators does not make their difference a direct measure of repair errors. A common-task analysis with explicit abstention outcomes is needed.
Questions for authors
- Does the claimed gap still widen when validity and restoration are measured on exactly the same tasks, with abstentions accounted for?
- Which experimental result isolates the benefit of versioning or producer attribution beyond the validation-and-repair stages already tested?