- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how many relation types a text-to-graph schema should contain and proposes SchemaGain. The method is a criterion that scores a candidate relation inventory by how much a frozen language model's surprisal on the corpus drops when the extracted triples are placed in context, minus the model's codelength for the list of relation names. The method starts from a single open extraction pass and keeps the 60 most frequent induced relation labels. It then greedily merges pairs of similar labels by relabeling the cached triples, and the size at which the score peaks is taken as the recommended inventory size. Experiments cover the DBpedia-WebNLG corpora of Text2KGBench. The authors report that every score curve has an interior maximum and that the selected size correlates with the size of the benchmark ontology (Pearson r = 0.70, mean absolute error of 7.3 types), which they read as agreement with expert judgment of granularity.
Critical strengths
- The paper proposes a smart way to decide the optimal schema size using a language model to score the Gain
- The provided results show a peak in the gain with varying size and also correlate with the gold size. Further comparisons with baselines show competitive F1 scores
Critical weaknesses
W1. The ground truth is not expert judgment of granularity. The paper calls the DBpedia-WebNLG sizes "fixed by human experts" (§1). Text2KGBench (§3.2) instead says each ontology was obtained by analysing the DBpedia triples in a WebNLG category and extracting the relations present, and the sentences are crowd-sourced verbalizations of those triples. The paper also reverses this order when it says the sentences were written from the ontologies' triples. So k_gold is the number of distinct predicates underlying each corpus, and the main result (r = 0.70) measures recovery of a count derived from the data. The expert-curated ontologies in Text2KGBench belong to the Wikidata-TekGen half, which the paper leaves out of quantitative evaluation.
W2. The Gain score itself is never validated: Only the location of its maximum is compared to an external quantity, and only as one number per corpus (n = 19, 95% CI [0.34, 0.90], leave-one-out r from 0.58 to 0.80). The paper never shows that schemas with higher Gain are better by any external measure, such as downstream utility or alignment with the gold relation partition. The interior peak in every curve (§5.1) is close to guaranteed by construction, since schema cost falls with every merge and the final forced merges collapse distinct relations. It does not count as independent evidence.
W3. App. F states the agreement is "in size and not in content." Only 48–57% of gold relations survive as their own type at the peak. Hitting the right count with a different partition is weak support for the claim of construct validity.
W4. One benchmark, with no held-out evaluation: All quantitative evidence comes from one benchmark family of short, fact-dense English sentences written from the reference triples. Every design choice was set and assessed on the same 19 corpora: the 60-label cap, the nearest-neighbour candidate pairs, the 2% plateau, and the F1 matching thresholds, which were replaced after the first grid underperformed. The cap-80 and cap-100 reruns that raise r to 0.80 were run after the main result was known (App. C). There is no study where the true number of relations is known and the text was not generated from the target ontology. As a result, it isn't clear whether the method finds an appropriate granularity or reverse-engineers the predicate count of a verbalization dataset.
W5. The baselines are weak and incomplete. The comparison covers only generic size heuristics applied to the same 60-label vocabulary.
- SchemaGain is not significantly better than threshold-HAC on error or correlation (Table 6).
- A hindsight constant and simple label counts nearly match its error.
- None of the schema-induction systems discussed in related work (EDC, KGGen, AutoSchemaKG, SAC-KG) is evaluated. The reason given for skipping EDC, a single-GPU budget, is not convincing for 7B-scale models.
- Also missing: open relation extraction methods that estimate the number of relations (e.g., PromptORE), LLM-guided granularity selection over a merge hierarchy (ClusterLLM), and classical model selection (BIC or a standard two-part MDL) on the same merge path.
W6. The motivation doesn't match the method. The paper is framed around a choice pipelines make "before any extraction happens," using a quantity "computable from the corpus alone." In practice k* requires a full open extraction of the corpus and greedy merging all the way down to one relation. It is then a post-hoc pick from the extractor's own capped vocabulary, so k* cannot include relations the extractor never produced; App. F reports 41% of gold relations lost before the search on the missed corpora. The authors concede that k* depends jointly on corpus, extractor and scorer (§3.3) and grows with corpus size (App. D). Nothing shows that a k* chosen on one sample transfers to unseen text from the same domain.
W7. The objective that is optimized is not the one stated. Eq. 1 is defined with schema-conditioned extraction. The search instead relabels cached triples from one open pass, which §5.5 itself shows produces different triples than extraction under a schema. The selected size is never checked by re-extracting under neighbouring sizes and recomputing Gain.
W8. The "no tuned trade-off" claim is overstated. The schema cost L(O) depends on how the label list is written: label names, the prefix sentence, separators and tokenization. That sets the effective balance between the two terms. The serialization ablation changes only how triples are written, not the schema. Since merging only renames labels, the benefit term mostly measures how much a label's wording hints at the sentence's words ("bornIn" helps predict "was born in"), not semantic structure.
W9. Downstream results don't support the selected size. Any schema beats schema-free extraction, but above about 10 types F1 is flat, and the unmerged 60-type schema has the best mapped F1 (Table 9: 0.217 vs 0.207). The routing benchmark gives the same curve with plain embedding clustering, and the authors say it "does not single out SchemaGain." The selection corpus of 4,860 sentences appears to include the 2,014 "held-out" test sentences.
W10. Novelty is limited and related work is missing. The approach applies MDL-style selection of a cluster count, using a language model's surprisal reduction as the fit term. That term resembles reference-free information measures such as the Shannon Score / BLANC family. The related-work section omits the closest prior lines:
- probabilistic relation induction from extracted tuples (Kok & Domingos 2008; Yao et al. 2011)
- open relation extraction with an estimated number of relations
- LLM-guided granularity selection (ClusterLLM)
- information-retention evaluation of extracted graphs (KGGen's MINE)
- natural-language parameterized models fit by likelihood (Zhong et al. 2024)
The claim of "the first computable definition of the size a relation inventory" should be tempered.
References
[1] Stanley Kok and Pedro Domingos. 2008. Extracting Semantic Networks from Text via Relational Clustering. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2008), Lecture Notes in Artificial Intelligence, vol. 5211, pp. 624–639. Springer.
[2] Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011. Structured Relation Discovery using Generative Models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1456–1466, Edinburgh, Scotland, UK.
[3] Pierre-Yves Genest, Pierre-Edouard Portier, Előd Egyed-Zsigmond, and Laurent-Walter Goix. 2022. PromptORE: A Novel Approach Towards Fully Unsupervised Relation Extraction. arXiv:2304.01209.
[4] Jun Zhao, Yongxin Zhang, Qi Zhang, Tao Gui, Zhongyu Wei, Minlong Peng, and Mingming Sun. 2023. Actively Supervised Clustering for Open Relation Extraction. arXiv:2306.04968.
[5] Belinda Mo, Kyssen Yu, Joshua Kazdan, Joan Cabezas, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, and Sanmi Koyejo. 2025. KGGen: Extracting Knowledge Graphs from Plain Text with Language Models. In NeurIPS. arXiv:2502.09956.
[6] Ruiqi Zhong, Heng Wang, Dan Klein, and Jacob Steinhardt. 2024. Explaining Datasets in Words: Statistical Models with Natural Language Parameters. arXiv:2409.08466
[7] Nicholas Egan, Oleg Vasilyev, and John Bohannon. 2022. Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22).
Questions for authors
- Given that Text2KGBench built each DBpedia-WebNLG ontology by extracting the relations present in that category's triples, how do you justify calling these sizes expert judgments of granularity? How does SchemaGain do against an independently designed ontology?
- Can you report results on corpora where the true number of relations is known and the text wasn't written from the target ontology? For example, subsets of k sampled FewRel relations for varying k, or a synthetic corpus with planted synonym labels.
- Validating Gain. Along a trajectory, what is the correlation between Gain and an external quality measure (downstream F1, routing F1, partition agreement)? For two schemas of equal size, is the higher-Gain one closer to the gold relations?
- Partition quality. What are ARI/NMI (or relation-level precision/recall) between the induced partition at k* and the gold relations, for SchemaGain and each baseline?
- Transfer. If k* is selected on one half of a corpus, how well does it fit the other half? How does k* change when the corpus is subsampled or doubled?
- Schema-conditioned check. For a few corpora, can you re-extract under schemas at and around k* and recompute Gain, to show the relabeling approximation preserves the location of the peak?
- Schema serialization. How sensitive is k* to how the relation list is written (label casing, separators, prefix, adding domain and range), and to an explicit weight on L(O)? What does Gain without the schema-cost term select?
- Baselines. Why weren't EDC, KGGen, AutoSchemaKG or SAC-KG run, even with smaller or quantized models? How does SchemaGain compare to PromptORE-style estimation of k, ClusterLLM's granularity selection, and BIC or a standard two-part MDL on the same merge path?
- Post-hoc choices. Which settings were fixed before seeing results on the 19 corpora? Can you report results with all post-hoc changes (cap, thresholds) reverted, or on a held-out benchmark?
- Test overlap: Were the 2,014 test sentences used for downstream F1 part of the 4,860-sentence corpus on which k* was selected?
- Why does the unmerged 60-type schema score the best mapped F1 in Table 9? What practical benefit does choosing k* give over that default?
- How does the method behave on naturally written corpora with fixed schemas, such as TACRED, Re-DocRED, SciERC or CoNLL04?