acceptodds
Under review as a conference paper at ICLR 2027

EviOntoKG: Paper-Specific Ontology Induction for Evidence-Grounded Multimodal Scientific Knowledge Graph Construction

Abstract

Constructing scientific knowledge graphs from academic papers is challenging because critical evidence is distributed across sections, tables, and formulas, and the meaning of individual claims often depends on the overall logic of the paper. Existing methods typically rely on schemas shared across papers or organize semantic structure only after extraction, making it difficult to adapt knowledge extraction to the specific context of each paper. We propose EviOntoKG, a two-scale framework for paper-level scientific knowledge graph construction. Inspired by the way researchers read scientific papers, EviOntoKG combines fast reading to reconstruct a paper-wide research chain with slow reading to analyze local multimodal evidence from text, tables, and formulas, thereby inducing a paper-specific ontology that is finalized before extraction to guide modality-specific extractors. On a dataset of 720 scientific papers, EviOntoKG achieves the best performance across all nine graph-quality metrics and all five downstream QA metrics, including 0.968 claim reliability, 0.741 source coverage, and 0.458 supported QA, the latter representing a 20.6% relative improvement over the best-performing baseline; it also achieves the highest cross-modal correspondence among all evaluated methods. Ablation studies further demonstrate the effectiveness of the major design choices in EviOntoKG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge Graph Construction, Scientific Document Understanding, Multimodal Information Extraction, Ontology Induction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mkY3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

EviOntoKG transforms scientific papers into evidence-grounded knowledge graphs by deriving paper-specific ontologies from their research logic and using them to guide knowledge extraction.

Critical strengths

The research motivation and framework workflow are reasonably clear.

Critical weaknesses
  1. The equations formalize the workflow but offer limited insight into the mechanisms underlying key operations, diminishing the value of the formalization.
  2. The paper lacks extraction precision, recall, and F1 scores against an independent, human-annotated gold standard. Its LLM-based validity and coverage metrics, despite judge agreement checks, provide limited evidence of extraction accuracy and completeness. There are concerns about the reliability of the experimental results.
  3. The figures are too small to read clearly.
Questions for authors
  1. How much do these research-specific designs improve upon a general document extraction framework with section analysis, dynamic ontologies, and evidence verification? Current ablations assess some components but do not isolate the contribution of research specificity itself.
  2. Are the datasets and implementation code for the baseline methods available?
Reviewer_ZcHVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The subject of this paper is automatic knowledge graph extraction from scientific papers. The approach follows the standard template: a user-defined schema is provided, the paper is segmented, and extraction is applied to each segment. The claimed novelties are: (i) an additional step in which the given schema is enriched by schema items extracted from the paper itself (termed the paper-specific ontology) and frozen before extraction; (ii) two passes of extraction: a preliminary pass per section that labels the section's rhetorical role and performs an open-vocabulary extraction with quotes (the paper's "fast reading", whose outputs are the SectionCards and a paper-level "research chain", and which also supplies the candidates for the paper-specific ontology), and the main pass, which is standard schema-constrained extraction with the output of the preliminary pass as additional input.

The quality of the resulting knowledge graph is evaluated on 720 arXiv papers, mainly via LLM-judge based metrics.

Critical strengths

Knowledge graph extraction from scientific papers is an important problem, and the pipeline is a substantial piece of engineering that treats tables and formulas as evidence with cell-level and region-level locators, which none of the compared systems does.

Critical weaknesses

There are serious issues with novelty and evaluation quality.

Novelty. The proposed improvements, the paper-specific ontology and the section cards (which also perform a preliminary extraction), are incremental modifications of the standard pipeline. Moreover, variants of both types of improvement exist in the literature, and there are published methods that employ both. See the details below.

Evaluation.

  1. No human-labelled measurement of any system. Every quality number is one LLM's verdict. The 450 human annotations were spent calibrating the judge, not comparing methods, and which methods' outputs they covered is unstated. Every published baseline the paper runs was evaluated in its own publication against gold labels, human judgment, or human-written downstream benchmarks.

  2. The judge is the builder's family, and the cross-family check is withheld. The graph is extracted by a DeepSeek model of unstated version and scored by DeepSeek-v4-pro, and the paper does not say which model runs its build-time grounding, validation and checklist agents. The GPT-6-luna re-evaluation was run on all methods (Appendix C.4), and only the method's own values and rank correlations appear, with the method's VSR falling from 0.968 to about 0.86 (Figure 3b), more than its gap to every baseline above 0.86.

  3. The comparison is not token matched. The method spends 1.34 M tokens per paper against about 50,000 for the nearest baseline on VSR (Schema-Miner + DeepSeek) and 23,000 for the plain prompt (Table 10, appendix only). No budget-matched baseline exists.

  4. No variance is reported, and the effects are inside the noise. Single runs everywhere. The full model appears at four VSR values spanning 0.06 across its own tables (0.968, 0.945, 0.916, 0.908). Every ablation effect on VSR that the paper discusses is smaller than that.

For instance, the full pipeline appears in Table 1 with VSR 0.968 (on 720 papers), but in Table 16 with VSR 0.945 (on a subsample of 72 papers), and in Table 11, with the repair pass included, at 0.908. Taking into account that baselines reach VSR 0.941 and 0.931 in Table 1, it is not clear that the reported improvement is real and not a random fluctuation.

In more detail on novelty. Where each component already exists, and what the paper's own ablations measure for it.

  1. Schema established before extraction. Extraction into a supplied schema is the standard setting (OneKE, a baseline here, is schema-guided by name). Inducing a schema and then extracting under it is EDC+R (Zhang and Soh, 2024, a baseline here): open extraction, schema definition and canonicalization, then a second extraction pass under the induced schema with the first pass's entities and relations as hints. The paper describes EDC as post hoc on the page where it names "the stage at which semantic structure enters the pipeline" as its novelty axis, and Table 5 lists EDC without the refinement pass. iText2KG (Lairgi et al., 2024) applies a predefined per-document blueprint before extraction. The paper's own DeepSeek Direct baseline prompt reads "first build a paper-specific ontology, then extract an ontology-aligned knowledge graph". What is specific here is that a fixed scholarly schema is specialised per paper by one LLM call and then frozen. Table 8 shows that the fixed schema's terms dominate the output (method in 96 % of papers, proposes in 89 %), and Figure 4a prices the per-paper layer at +0.015 VSR over applying the same ontology after extraction and +0.027 over the fixed schema alone, in single runs with paper counts unstated.

  2. Two-pass, section-aware extraction. Labelling sections by rhetorical role before extraction is a long-standing device in scientific-document processing, and a first LLM pass whose output guides a second is EDC+R's refinement and iText2KG's document distiller. AutoQG (Shen et al., 2026), which the paper cites as the source of its QA questions, already performs section-aware extraction with per-section prompts by discourse role into triples with evidence locators under a scholarly schema (models, datasets, metrics, methods; evaluatedOn, proposes, addresses), with an LLM evaluation agent scoring Domain Fit, Accuracy, Consistency, Completeness and Granularity and an iterate-until-threshold refinement loop. The submission's judge rubric uses the same five dimensions under slightly different names, and its baseline prompts carry AutoQG's role line and contextText field, but the related work does not mention AutoQG and it is not run. Figure 4b prices the research chain at −0.015 VSR and −0.030 SCR.

  3. Table and formula channels. The parsing is MinerU (cited) and the formula schema is FORCE (cited); the integration is the paper's.

  4. Admission gates and refinement. The evaluate-and-refine loop with a weighted quality score is AutoQG's evaluation agent. The paper's own appendix shows that the refinement pass lowers VSR and all five quality dimensions (Table 12) and that removing any of the four admission gates leaves the final graph unchanged (Table 16). The main text reports neither.

Questions for authors

Please discuss the items in the weakness section.

Reviewer_yjFKOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EVIONTOKG for constructing scientific knowledge graphs from text, tables, and formulas. The main idea is to induce a paper-specific ontology before claim extraction. The authors evaluate the method on 720 arXiv papers and compare it with existing KG construction methods.

Critical strengths
  • The paper studies an interesting problem about constructing evidence-grounded KGs from scientific papers. The idea of inducing an ontology before extraction is reasonable and has practical value.

  • The proposed framework is well designed. Specifically, the frozen ontology and ClaimRecord provide a structured way to connect entities, relations, and evidence from different modalities. This is useful for keeping the source information during KG construction.

  • The authors conduct several ablation studies to verify the effectiveness of different components. Figure 4 and Tables 14–15 provide useful analysis about ontology induction, structural guidance, and multimodal information.

Critical weaknesses
  • The comparison with baseline methods is not fully fair. Specifically, some baselines do not support evidence binding and receive an ETR score of 0 (Tables 1 and 5). Therefore, it is difficult to determine whether the improvement comes from better extraction ability or simply different representation settings.

  • Some baseline implementations are modified from the original methods. For example, the authors replace the trained T5 pruner in SAC-KG with a DeepSeek classifier (Appendix B.1). However, such modifications may affect the performance of baselines and make the comparison less convincing.

  • The computational cost is not well controlled. The authors do not provide comparison under the same model and token budget. As a result, we cannot clearly know how much improvement comes from the proposed framework rather than more computational resources.

  • I have concerns about the reliability of evaluation results. The authors mainly use automatic metrics to evaluate KG quality, and only sample at most 12 claims per paper (Appendix C.1). Although human calibration is provided, it does not directly verify the system-level comparison. More human evaluation is needed to support the results.

  • The downstream QA evaluation is not sufficiently validated. The reference answers are imported from AutoQG, but the authors do not clearly report independent human verification. Therefore, it remains unclear whether the improvement in QA performance really reflects better KG quality.

  • The method is too engineering-oriented. It looks more like a combination of existing techniques rather than a new technical contribution. The authors should better clarify the novelty beyond integrating these components.

Detailed Review: This was a frustrating reading. I spent considerable time trying to understand what the authors actually propose, what is new, and why the proposed approach should work. The paper introduces many concepts and components, but the explanations are often abstract, and I found myself repeatedly asking rather basic questions that I would expect the methodology to answer. Here is my updated review, with additional details on my concerns.

My first difficulty is understanding what the authors mean by a paper-specific ontology and, especially, a local ontology. The paper starts with reusable scholarly and formula ontologies, introduces local ontologies during slow reading, and eventually produces a frozen paper-specific ontology. But what exactly distinguishes these objects? Is a local ontology simply a collection of types and relations identified in one section? How is that different from the Local Patterns discussed in Section 3? How are these local ontologies merged into the paper-specific ontology? What happens when two sections introduce overlapping or conflicting concepts? And what exactly is inherited from the reusable global ontology, as opposed to being learned from the paper?

I found the discussion of hard constraints and soft guidance equally difficult to follow. The introduction suggests that the frozen paper-specific ontology and its local ontologies provide hard constraints, while the Research Chain and section roles provide soft guidance. Section 3, however, treats Local Patterns as soft guidance. Are these Local Patterns the same as the local ontologies? If not, what distinguishes them? What makes the constraints hard? Section 3.5 allows typed extensions that do not directly match the frozen ontology. If such extensions are permitted, what exactly is frozen, and how are these exceptions controlled?

I believe a concrete example would go much further than the current descriptions. Take one scientific paper. Show me the initial ontology, the concepts and relations identified in two different sections, the resulting local ontologies, and how they are combined into the final paper-specific ontology. Which types and relations were inherited? Which were newly discovered? Which were merged or rejected, and why? Then show how the resulting ontology changes the extraction of a particular claim. This would make the proposed contribution much easier to understand.

Figure 2 is useless for human consumption. It is so crowded with boxes, arrows, and tiny text that it is practically unreadable at normal viewing size. It looks more like a diagram generated to include every implementation detail than a figure designed to explain a research idea. What is a reader supposed to learn from it?

I also have difficulty identifying the methodological contribution of Section 3. The framework parses documents, summarizes sections, identifies research concepts and relationships, creates an ontology, extracts claims, validates them, and assembles a knowledge graph. These are all reasonable steps. But which of them constitutes a new research contribution? Is there a new method for ontology induction? A new extraction algorithm? A new way of reconciling ontologies? A new validation technique? Or is the contribution primarily the arrangement of existing LLM-based components into a pipeline? There is nothing inherently wrong with building an integrated system, but then the paper needs to explain what makes this particular integration interesting and what insight it offers beyond using multiple LLM calls and increasingly detailed prompts.

This leads to my next concern: why should the proposed method work? The authors motivate fast and slow reading through an analogy with how researchers read scientific papers. But an appealing analogy is not an explanation of an algorithm. Why should reconstructing a Research Chain improve ontology induction? Why is freezing the ontology before extraction better than allowing it to evolve as new evidence is encountered? Could freezing the ontology actually prevent the extraction of legitimate claims that were missed during the initial reading? What is the underlying reasoning that justifies these design decisions?

The connection between Section 3 and the experimental results also seems weak. I recognize that the paper contains many experiments and ablations. However, most evaluate the final knowledge graph rather than the intermediate structures that are supposedly responsible for its quality. How do we know that the Research Chain correctly represents the argument of a scientific paper? How do we know that the local ontologies are meaningful or correct? How much of the improvement comes from the induced ontology, as opposed to giving the extractor additional context and making more LLM calls?

I am also confused by the scientific expressibility metric in Table 2. What exactly does scientific expressibility mean in this setting? How is it different from semantic completeness? Appendix C.2 provides a definition of CE@1 based on whether a source proposition can be represented with a perfect score using the supplied ontology. But how are these source propositions selected, and what qualifies as a fully expressive representation? Why is a score of 10 required? What would a score of 8 represent in terms of scientific knowledge lost? More importantly, how should we interpret the fact that the proposed method achieves only 33.3% CE@1 and 33.5% strict semantic completeness? Does that mean approximately two-thirds of the sampled scientific requirements cannot be fully represented? I would expect these results to receive much more explanation, preferably with examples of successful and unsuccessful representations.

There is also relevant prior work that makes the novelty question more pressing. For example, Acharya et al., Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications (AACL 2026, https://aclanthology.org/2026.aacl-long.252/), already describe an agent-based framework for ontology-guided scientific knowledge graph construction, with evidence-grounded extraction, ontology-conformance checking, and multistage verification. I acknowledge that this paper appeared recently and that the authors may not have been aware of it when submitting their work. I am not raising its omission as a criticism in itself. However, it provides a useful point of comparison. What does EviOntoKG add to ontology-guided scientific knowledge extraction beyond existing constraint and verification mechanisms? The apparent distinction is the induction of a separate ontology for each paper, along with multimodal processing. If that is the central contribution, it needs to be demonstrated much more clearly, both methodologically and experimentally.

The improvements in graph quality and downstream question answering are interesting, and the authors have clearly invested effort in building and evaluating the system. My difficulty is connecting those results to a clearly identifiable research contribution. I finished Section 3 without an understanding of what is new in the individual components, and I finished the experiments without a convincing explanation of why the proposed approach works.

The paper reads more like an elaborate engineering exercise than a well-motivated methodological contribution. The authors need to explain the central ideas in concrete terms, demonstrate what is actually learned at the different ontology levels, and provide evidence that these ideas, rather than simply the complexity of the pipeline, account for the reported improvements.

I conclude with two other questions that marked my reading: If every paper has its own ontology, how do we construct a knowledge graph from thousands or millions of scientific papers?

Why do we need a separate ontology for each paper rather than a shared scholarly ontology with paper-specific instances, attributes, and qualifiers?

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.