acceptodds
Withdrawn from ICLR 2027

How Many Relation Types Do You Need? Defining a Corpus-Relative Optimum for Text-to-Graph Schemas

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Every text-to-graph pipeline commits to a schema before extraction, yet prior work offers no principled answer to how large that schema should be. We introduce SchemaGain, which scores a candidate relation inventory by the surprisal that its extracted graph removes from the corpus, minus the language-model cost of describing the schema itself. Across all 19 DBpedia-WebNLG corpora in Text2KGBench, SchemaGain has an interior optimum and aligns with expert-designed inventories (Pearson ; across seeds), while alternative selectors based on clustering, frequency, or direct LLM judgment show weaker agreement and greater variation across seeds. Randomly merging relations through otherwise identical machinery reduces the correlation to . On Text2KGBench's extraction task, using any schema improves F1 by 61% over schema-free extraction and reduces subject hallucination from 41% to below 1%. The cost of larger inventories becomes apparent downstream: in a routing benchmark over the same graphs, relation retrieval peaks near 35 types, and at 60 types a label router's top-1 accuracy is half that at 30. A full sweep takes about four hours on one consumer GPU. Together, these results suggest that schema size is a measurable modeling choice rather than an arbitrary design parameter.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
minimum description length, knowledge graph construction, schema induction, model selection, information extraction, large language models, compression, ontology learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.20 over 5 reviews, above 2% of the others.
Reviewer_ZMSBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how many relation types a text-to-graph schema should contain and proposes SchemaGain. The method is a criterion that scores a candidate relation inventory by how much a frozen language model's surprisal on the corpus drops when the extracted triples are placed in context, minus the model's codelength for the list of relation names. The method starts from a single open extraction pass and keeps the 60 most frequent induced relation labels. It then greedily merges pairs of similar labels by relabeling the cached triples, and the size at which the score peaks is taken as the recommended inventory size. Experiments cover the DBpedia-WebNLG corpora of Text2KGBench. The authors report that every score curve has an interior maximum and that the selected size correlates with the size of the benchmark ontology (Pearson r = 0.70, mean absolute error of 7.3 types), which they read as agreement with expert judgment of granularity.

Critical strengths
  1. The paper proposes a smart way to decide the optimal schema size using a language model to score the Gain
  2. The provided results show a peak in the gain with varying size and also correlate with the gold size. Further comparisons with baselines show competitive F1 scores
Critical weaknesses

W1. The ground truth is not expert judgment of granularity. The paper calls the DBpedia-WebNLG sizes "fixed by human experts" (§1). Text2KGBench (§3.2) instead says each ontology was obtained by analysing the DBpedia triples in a WebNLG category and extracting the relations present, and the sentences are crowd-sourced verbalizations of those triples. The paper also reverses this order when it says the sentences were written from the ontologies' triples. So k_gold is the number of distinct predicates underlying each corpus, and the main result (r = 0.70) measures recovery of a count derived from the data. The expert-curated ontologies in Text2KGBench belong to the Wikidata-TekGen half, which the paper leaves out of quantitative evaluation.

W2. The Gain score itself is never validated: Only the location of its maximum is compared to an external quantity, and only as one number per corpus (n = 19, 95% CI [0.34, 0.90], leave-one-out r from 0.58 to 0.80). The paper never shows that schemas with higher Gain are better by any external measure, such as downstream utility or alignment with the gold relation partition. The interior peak in every curve (§5.1) is close to guaranteed by construction, since schema cost falls with every merge and the final forced merges collapse distinct relations. It does not count as independent evidence.

W3. App. F states the agreement is "in size and not in content." Only 48–57% of gold relations survive as their own type at the peak. Hitting the right count with a different partition is weak support for the claim of construct validity.

W4. One benchmark, with no held-out evaluation: All quantitative evidence comes from one benchmark family of short, fact-dense English sentences written from the reference triples. Every design choice was set and assessed on the same 19 corpora: the 60-label cap, the nearest-neighbour candidate pairs, the 2% plateau, and the F1 matching thresholds, which were replaced after the first grid underperformed. The cap-80 and cap-100 reruns that raise r to 0.80 were run after the main result was known (App. C). There is no study where the true number of relations is known and the text was not generated from the target ontology. As a result, it isn't clear whether the method finds an appropriate granularity or reverse-engineers the predicate count of a verbalization dataset.

W5. The baselines are weak and incomplete. The comparison covers only generic size heuristics applied to the same 60-label vocabulary.

  • SchemaGain is not significantly better than threshold-HAC on error or correlation (Table 6).
  • A hindsight constant and simple label counts nearly match its error.
  • None of the schema-induction systems discussed in related work (EDC, KGGen, AutoSchemaKG, SAC-KG) is evaluated. The reason given for skipping EDC, a single-GPU budget, is not convincing for 7B-scale models.
  • Also missing: open relation extraction methods that estimate the number of relations (e.g., PromptORE), LLM-guided granularity selection over a merge hierarchy (ClusterLLM), and classical model selection (BIC or a standard two-part MDL) on the same merge path.

W6. The motivation doesn't match the method. The paper is framed around a choice pipelines make "before any extraction happens," using a quantity "computable from the corpus alone." In practice k* requires a full open extraction of the corpus and greedy merging all the way down to one relation. It is then a post-hoc pick from the extractor's own capped vocabulary, so k* cannot include relations the extractor never produced; App. F reports 41% of gold relations lost before the search on the missed corpora. The authors concede that k* depends jointly on corpus, extractor and scorer (§3.3) and grows with corpus size (App. D). Nothing shows that a k* chosen on one sample transfers to unseen text from the same domain.

W7. The objective that is optimized is not the one stated. Eq. 1 is defined with schema-conditioned extraction. The search instead relabels cached triples from one open pass, which §5.5 itself shows produces different triples than extraction under a schema. The selected size is never checked by re-extracting under neighbouring sizes and recomputing Gain.

W8. The "no tuned trade-off" claim is overstated. The schema cost L(O) depends on how the label list is written: label names, the prefix sentence, separators and tokenization. That sets the effective balance between the two terms. The serialization ablation changes only how triples are written, not the schema. Since merging only renames labels, the benefit term mostly measures how much a label's wording hints at the sentence's words ("bornIn" helps predict "was born in"), not semantic structure.

W9. Downstream results don't support the selected size. Any schema beats schema-free extraction, but above about 10 types F1 is flat, and the unmerged 60-type schema has the best mapped F1 (Table 9: 0.217 vs 0.207). The routing benchmark gives the same curve with plain embedding clustering, and the authors say it "does not single out SchemaGain." The selection corpus of 4,860 sentences appears to include the 2,014 "held-out" test sentences.

W10. Novelty is limited and related work is missing. The approach applies MDL-style selection of a cluster count, using a language model's surprisal reduction as the fit term. That term resembles reference-free information measures such as the Shannon Score / BLANC family. The related-work section omits the closest prior lines:

  • probabilistic relation induction from extracted tuples (Kok & Domingos 2008; Yao et al. 2011)
  • open relation extraction with an estimated number of relations
  • LLM-guided granularity selection (ClusterLLM)
  • information-retention evaluation of extracted graphs (KGGen's MINE)
  • natural-language parameterized models fit by likelihood (Zhong et al. 2024)

The claim of "the first computable definition of the size a relation inventory" should be tempered.

References

[1] Stanley Kok and Pedro Domingos. 2008. Extracting Semantic Networks from Text via Relational Clustering. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2008), Lecture Notes in Artificial Intelligence, vol. 5211, pp. 624–639. Springer.

[2] Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011. Structured Relation Discovery using Generative Models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1456–1466, Edinburgh, Scotland, UK.

[3] Pierre-Yves Genest, Pierre-Edouard Portier, Előd Egyed-Zsigmond, and Laurent-Walter Goix. 2022. PromptORE: A Novel Approach Towards Fully Unsupervised Relation Extraction. arXiv:2304.01209.

[4] Jun Zhao, Yongxin Zhang, Qi Zhang, Tao Gui, Zhongyu Wei, Minlong Peng, and Mingming Sun. 2023. Actively Supervised Clustering for Open Relation Extraction. arXiv:2306.04968.

[5] Belinda Mo, Kyssen Yu, Joshua Kazdan, Joan Cabezas, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, and Sanmi Koyejo. 2025. KGGen: Extracting Knowledge Graphs from Plain Text with Language Models. In NeurIPS. arXiv:2502.09956.

[6] Ruiqi Zhong, Heng Wang, Dan Klein, and Jacob Steinhardt. 2024. Explaining Datasets in Words: Statistical Models with Natural Language Parameters. arXiv:2409.08466

[7] Nicholas Egan, Oleg Vasilyev, and John Bohannon. 2022. Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22).

Questions for authors
  1. Given that Text2KGBench built each DBpedia-WebNLG ontology by extracting the relations present in that category's triples, how do you justify calling these sizes expert judgments of granularity? How does SchemaGain do against an independently designed ontology?
  2. Can you report results on corpora where the true number of relations is known and the text wasn't written from the target ontology? For example, subsets of k sampled FewRel relations for varying k, or a synthetic corpus with planted synonym labels.
  3. Validating Gain. Along a trajectory, what is the correlation between Gain and an external quality measure (downstream F1, routing F1, partition agreement)? For two schemas of equal size, is the higher-Gain one closer to the gold relations?
  4. Partition quality. What are ARI/NMI (or relation-level precision/recall) between the induced partition at k* and the gold relations, for SchemaGain and each baseline?
  5. Transfer. If k* is selected on one half of a corpus, how well does it fit the other half? How does k* change when the corpus is subsampled or doubled?
  6. Schema-conditioned check. For a few corpora, can you re-extract under schemas at and around k* and recompute Gain, to show the relabeling approximation preserves the location of the peak?
  7. Schema serialization. How sensitive is k* to how the relation list is written (label casing, separators, prefix, adding domain and range), and to an explicit weight on L(O)? What does Gain without the schema-cost term select?
  8. Baselines. Why weren't EDC, KGGen, AutoSchemaKG or SAC-KG run, even with smaller or quantized models? How does SchemaGain compare to PromptORE-style estimation of k, ClusterLLM's granularity selection, and BIC or a standard two-part MDL on the same merge path?
  9. Post-hoc choices. Which settings were fixed before seeing results on the 19 corpora? Can you report results with all post-hoc changes (cap, thresholds) reverted, or on a held-out benchmark?
  10. Test overlap: Were the 2,014 test sentences used for downstream F1 part of the 4,860-sentence corpus on which k* was selected?
  11. Why does the unmerged 60-type schema score the best mapped F1 in Table 9? What practical benefit does choosing k* give over that default?
  12. How does the method behave on naturally written corpora with fixed schemas, such as TACRED, Re-DocRED, SciERC or CoNLL04?
Reviewer_CuhGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses how to select the number of suitable relations that a text-to-graph approach should extract: if too little, the relations are generic; if too many, the relations are over-specific. The main approach takes is inspired by MDL, and uses model log-likelihoods to determine the description length of the model (= relations) and data (= text). The approach is evaluated on a benchmark with multiple datasets, each dataset consisting of verbalized facts from a "gold" knowledge graph.

Critical strengths

Simple approach, which is great.

Works with any LLM (as long as its output probabilities can be accessed).

Gives useful results in the experimental study.

Refreshing presentation. Clear, knowledgeable exposition, no unnecessary hyperbole or marketing. If only all papers were like this.

Critical weaknesses

Better fit to an NLP venue. The problem and approach is more suited to an NLP or KB construction venue; there is little novelty w.r.t. learning ("apply MDL to this task").

Heuristic approach. The actual method does not only decide on schema size, but also on the actual relations. The paper uses a simple, reasonable heuristic here, but it's not really a principled approach.

No theoretical analysis so that key properties of the approach remains unclear. For example, it appears that running the approach on dataset "D" and running it on "D union D" (each example replicated twice) produces different results, which does not appear to be desirable.

The experimental study is a useful first insight, but I am not sufficiently convinced by it. The benchmark appears to be a sweet spot for the proposed method, as it's a verbalized graph. There are no experiments on real text or actual tasks, and it's not clear how useful the approach would be in practice.

Finally, I'd expect that any fully automatic, task-agnostic method may be problematic: the "right" granularity (and the "right" vocabulary, too) depends on what one wants to do with the extracted facts, i.e., on the actual application. If so, the choice should be informed by the downstream task.

Questions for authors
Reviewer_NDnUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats schema size (the number of relation types) as a model-selection problem. It defines SchemaGain = L0 − L(O) − L(C|O), with every term measured by one frozen LM. It traces the objective along a greedy merge trajectory and shows that the peak size correlates with expert ontology sizes on Text2KGBench DBpedia-WebNLG.

Critical strengths
  • S1. A crisp problem statement (Eq. 1). Each limitation is stated where it applies (§3.1 approximation, §3.3 missing L(G|O) term, §6 scope).
  • S2. Honest statistics: bootstrap CIs, paired tests (Table 6), leave-one-out (App. C). The authors report where they cannot separate their method from baselines.
  • S3. Cheap and reproducible (one RTX 4080, about 4 h main sweep). Config-embedded result files are promised.
  • S4. The chunk-only scoring unit test (App. A) is a good practice.
Critical weaknesses
  • W1. SchemaGain assumes the gain measures how informative the schema's structure is. But relation labels are natural-language strings, so they can make the sentence less surprising to the scoring LM simply by lexically priming its words. L(O) likewise reflects how common the label strings are, not how many distinctions there are. Under LM likelihood, structural information and label-text effects are entangled and cannot be measured separately, so the claim that SchemaGain defines how many relation types a corpus needs does not follow. The authors should show that the results survive with opaque or paraphrased labels.
  • W2. The abstract and introduction claim more than the appendix supports. The abstract says alternative selectors "show weaker agreement and greater variation across seeds". Table 6 shows that the error and correlation differences from threshold-HAC under resampling both have intervals that include zero, and that the hindsight constant matches SchemaGain's MAE. Table 5 shows count-only selectors with smaller seed ranges than SchemaGain (see R1-M2).
  • W3. "To our knowledge, SchemaGain is the first computable definition of the size a relation inventory should have" (§1) is too strong given the nonparametric-Bayesian relation-discovery literature (see R2-M1).
  • W4. The abstract highlights "+61% F1" and "hallucination 41% → <1%". These results come from having a schema at all. Threshold-HAC's schema matches them (Table 8). Placed next to the method's claims, they invite misreading.
Questions for authors

no more question

Reviewer_x7VROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to determine the optimal number of relation types in Text2KG. The authors propose a metric called SchemaGain and evaluate it on Text2KGBench. The results show that SchemaGain can estimate an optimal number of relation types that is largely consistent with the number manually selected by human experts.

Critical strengths
  1. This paper attempts to quantify the optimal number of relation types in Text2KG, which is a meaningful research direction.
  2. The proposed method is relatively simple and does not introduce many hyperparameters.
  3. The experimental pipeline is clearly presented and exhibits good reproducibility.
Critical weaknesses
  1. The analysis is primarily conducted on pre-extracted triples by only changing their relation types. However, the schema design itself can influence the triple extraction process. Therefore, the current experimental setting may not accurately reflect the impact of relation type selection on end-to-end Text2KG performance.
  2. Most experiments are conducted on WebNLG from Text2KGBench. However, WebNLG consists of texts manually constructed based on pre-existing triples, which differs substantially from real-world open-domain knowledge extraction scenarios. Therefore, the conclusions drawn from this dataset may not be sufficiently convincing.
  3. The main conclusion is that both too few and too many relation types lead to performance degradation. However, this observation is already widely acknowledged in ontology engineering and knowledge graph construction. The paper does not sufficiently demonstrate what novel insights it provides beyond this established understanding.
  4. The experimental corpora mainly contain only a few hundred sentences, whereas practical applications such as GraphRAG often need to process millions of sentences. The effectiveness and stability of the proposed method on large-scale corpora remain unclear.
  5. The paper lacks comparisons with existing model selection and information-theoretic criteria, such as BIC and MML.
  6. OpenIE has already explored relation clustering, normalization, and merging to reduce redundant relation types. The paper should better clarify how its approach differs from these existing methods.
Questions for authors
  1. In Table 2, the correlation between Sentences and Peak does not appear to be particularly strong. Could the authors explain this observation?
  2. The schema induction pipeline relies on heuristic rules at multiple stages. How do the authors justify these heuristics, and have their effectiveness and robustness been sufficiently validated?
  3. The selection of relation type cardinality and ontology granularity has long been studied in ontology engineering. What are the fundamental differences between this work and existing research on Ontology Learning and Schema Induction? What new insights does this paper provide?
Reviewer_HRHdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SchemaGain, a method for selecting the number of relation types in text-to-graph schemas. Inspired by the Minimum Description Length principle, SchemaGain balances the information gained from extracted graphs against the description cost of the schema. Experiments on 19 Text2KGBench corpora demonstrate that the selected schema sizes correlate with expert-designed ontologies and outperform several baseline selectors.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem that has received limited attention in existing text-to-graph research.

  2. The proposed information-theoretic formulation provides an interesting perspective on schema-size selection.

  3. The paper includes extensive experiments, baseline comparisons, and robustness analyses.

Critical weaknesses
  1. The paper primarily evaluates SchemaGain by measuring its agreement with expert-designed schema sizes. However, expert-defined relation inventories may reflect subjective modeling choices rather than optimal schema sizes for downstream applications. The authors should provide stronger justification for this evaluation criterion.
  2. The proposed method uses greedy merging and relabels cached extracted triples instead of performing schema-conditioned extraction for each candidate schema. However, different schemas may lead to different extracted triples, not merely different relation labels. It remains unclear how this approximation affects the selected schema size.
  3. Although the paper demonstrates that schema size affects downstream performance, it remains unclear whether maximizing SchemaGain leads to the optimal schema size for downstream tasks. Could the authors analyze the correlation between SchemaGain's selected sizes and the downstream-optimal sizes across different corpora?
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.