-
The comparison with baseline methods is not fully fair. Specifically, some baselines do not support evidence binding and receive an ETR score of 0 (Tables 1 and 5). Therefore, it is difficult to determine whether the improvement comes from better extraction ability or simply different representation settings.
-
Some baseline implementations are modified from the original methods. For example, the authors replace the trained T5 pruner in SAC-KG with a DeepSeek classifier (Appendix B.1). However, such modifications may affect the performance of baselines and make the comparison less convincing.
-
The computational cost is not well controlled. The authors do not provide comparison under the same model and token budget. As a result, we cannot clearly know how much improvement comes from the proposed framework rather than more computational resources.
-
I have concerns about the reliability of evaluation results. The authors mainly use automatic metrics to evaluate KG quality, and only sample at most 12 claims per paper (Appendix C.1). Although human calibration is provided, it does not directly verify the system-level comparison. More human evaluation is needed to support the results.
-
The downstream QA evaluation is not sufficiently validated. The reference answers are imported from AutoQG, but the authors do not clearly report independent human verification. Therefore, it remains unclear whether the improvement in QA performance really reflects better KG quality.
-
The method is too engineering-oriented. It looks more like a combination of existing techniques rather than a new technical contribution. The authors should better clarify the novelty beyond integrating these components.
Detailed Review:
This was a frustrating reading. I spent considerable time trying to understand what the authors actually propose, what is new, and why the proposed approach should work. The paper introduces many concepts and components, but the explanations are often abstract, and I found myself repeatedly asking rather basic questions that I would expect the methodology to answer. Here is my updated review, with additional details on my concerns.
My first difficulty is understanding what the authors mean by a paper-specific ontology and, especially, a local ontology. The paper starts with reusable scholarly and formula ontologies, introduces local ontologies during slow reading, and eventually produces a frozen paper-specific ontology. But what exactly distinguishes these objects? Is a local ontology simply a collection of types and relations identified in one section? How is that different from the Local Patterns discussed in Section 3? How are these local ontologies merged into the paper-specific ontology? What happens when two sections introduce overlapping or conflicting concepts? And what exactly is inherited from the reusable global ontology, as opposed to being learned from the paper?
I found the discussion of hard constraints and soft guidance equally difficult to follow. The introduction suggests that the frozen paper-specific ontology and its local ontologies provide hard constraints, while the Research Chain and section roles provide soft guidance. Section 3, however, treats Local Patterns as soft guidance. Are these Local Patterns the same as the local ontologies? If not, what distinguishes them? What makes the constraints hard? Section 3.5 allows typed extensions that do not directly match the frozen ontology. If such extensions are permitted, what exactly is frozen, and how are these exceptions controlled?
I believe a concrete example would go much further than the current descriptions. Take one scientific paper. Show me the initial ontology, the concepts and relations identified in two different sections, the resulting local ontologies, and how they are combined into the final paper-specific ontology. Which types and relations were inherited? Which were newly discovered? Which were merged or rejected, and why? Then show how the resulting ontology changes the extraction of a particular claim. This would make the proposed contribution much easier to understand.
Figure 2 is useless for human consumption. It is so crowded with boxes, arrows, and tiny text that it is practically unreadable at normal viewing size. It looks more like a diagram generated to include every implementation detail than a figure designed to explain a research idea. What is a reader supposed to learn from it?
I also have difficulty identifying the methodological contribution of Section 3. The framework parses documents, summarizes sections, identifies research concepts and relationships, creates an ontology, extracts claims, validates them, and assembles a knowledge graph. These are all reasonable steps. But which of them constitutes a new research contribution? Is there a new method for ontology induction? A new extraction algorithm? A new way of reconciling ontologies? A new validation technique? Or is the contribution primarily the arrangement of existing LLM-based components into a pipeline? There is nothing inherently wrong with building an integrated system, but then the paper needs to explain what makes this particular integration interesting and what insight it offers beyond using multiple LLM calls and increasingly detailed prompts.
This leads to my next concern: why should the proposed method work? The authors motivate fast and slow reading through an analogy with how researchers read scientific papers. But an appealing analogy is not an explanation of an algorithm. Why should reconstructing a Research Chain improve ontology induction? Why is freezing the ontology before extraction better than allowing it to evolve as new evidence is encountered? Could freezing the ontology actually prevent the extraction of legitimate claims that were missed during the initial reading? What is the underlying reasoning that justifies these design decisions?
The connection between Section 3 and the experimental results also seems weak. I recognize that the paper contains many experiments and ablations. However, most evaluate the final knowledge graph rather than the intermediate structures that are supposedly responsible for its quality. How do we know that the Research Chain correctly represents the argument of a scientific paper? How do we know that the local ontologies are meaningful or correct? How much of the improvement comes from the induced ontology, as opposed to giving the extractor additional context and making more LLM calls?
I am also confused by the scientific expressibility metric in Table 2. What exactly does scientific expressibility mean in this setting? How is it different from semantic completeness? Appendix C.2 provides a definition of CE@1 based on whether a source proposition can be represented with a perfect score using the supplied ontology. But how are these source propositions selected, and what qualifies as a fully expressive representation? Why is a score of 10 required? What would a score of 8 represent in terms of scientific knowledge lost? More importantly, how should we interpret the fact that the proposed method achieves only 33.3% CE@1 and 33.5% strict semantic completeness? Does that mean approximately two-thirds of the sampled scientific requirements cannot be fully represented? I would expect these results to receive much more explanation, preferably with examples of successful and unsuccessful representations.
There is also relevant prior work that makes the novelty question more pressing. For example, Acharya et al., Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications (AACL 2026, https://aclanthology.org/2026.aacl-long.252/), already describe an agent-based framework for ontology-guided scientific knowledge graph construction, with evidence-grounded extraction, ontology-conformance checking, and multistage verification. I acknowledge that this paper appeared recently and that the authors may not have been aware of it when submitting their work. I am not raising its omission as a criticism in itself. However, it provides a useful point of comparison. What does EviOntoKG add to ontology-guided scientific knowledge extraction beyond existing constraint and verification mechanisms? The apparent distinction is the induction of a separate ontology for each paper, along with multimodal processing. If that is the central contribution, it needs to be demonstrated much more clearly, both methodologically and experimentally.
The improvements in graph quality and downstream question answering are interesting, and the authors have clearly invested effort in building and evaluating the system. My difficulty is connecting those results to a clearly identifiable research contribution. I finished Section 3 without an understanding of what is new in the individual components, and I finished the experiments without a convincing explanation of why the proposed approach works.
The paper reads more like an elaborate engineering exercise than a well-motivated methodological contribution. The authors need to explain the central ideas in concrete terms, demonstrate what is actually learned at the different ontology levels, and provide evidence that these ideas, rather than simply the complexity of the pipeline, account for the reported improvements.
I conclude with two other questions that marked my reading:
If every paper has its own ontology, how do we construct a knowledge graph from thousands or millions of scientific papers?
Why do we need a separate ontology for each paper rather than a shared scholarly ontology with paper-specific instances, attributes, and qualifiers?