- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper focuses on Mathematical Knowledge Consolidation (MKC), which aims to construct consolidated mathematical knowledge graphs from mathematical documents. Specifically, it proposes Gangmu, a two-stage framework that first constructs an Evidence Graph and then canonicalizes mathematical statements through entity resolution. The paper also introduces MathER-Alg, a benchmark for mathematical entity resolution. The experimental results demonstrate the effectiveness of the proposed method on the benchmark.
The paper proposes a practical framework for mathematical knowledge consolidation and introduces a potentially useful benchmark for mathematical entity resolution. However, the experimental evaluation is insufficient to establish the superiority of the proposed framework over existing methods, particularly for the complete MKC task. Overall, this reviewer leans toward a negative score.
Critical strengths
- The two-stage framework, which separates mathematical knowledge extraction from entity resolution and canonicalization, is practical and reasonably designed.
- The introduced mathematical entity resolution benchmark may serve as a useful resource for future research.
- The methodology and experimental settings are described in sufficient detail.
Critical weaknesses
- The paper evaluates the two stages separately, namely Evidence Graph construction and mathematical entity resolution. However, the effectiveness of the complete MKC framework remains unclear, as the quality of the final Canonical Graph is not directly evaluated.
- The comparisons with existing methods are insufficient. First, the paper does not compare Gangmu with existing mathematical knowledge graph construction methods or general-purpose knowledge graph construction and consolidation methods adapted to mathematical documents. Second, the MathER evaluation mainly includes retrieval-based baselines and a general-purpose agent, without comparisons against established entity resolution methods. These missing comparisons make it difficult to assess the advantages of the proposed framework over existing techniques.
- The discussion of mathematics-specific challenges is insufficient. It remains unclear what makes these challenges fundamentally different in the mathematical domain and why existing general-purpose methods would be inadequate.
Reference:
[1] Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction
Questions for authors
None
5 comments
W2 (continued)
- KGGen (general-purpose knowledge graph construction): This method extracts entity–relation triples from text and consolidates the graph through aggregation, embedding-based clustering, and LLM-based deduplication. Using the official code, we retain these core steps and the original graph representation, making the necessary adaptations and fixes to the model interface, provenance recording, and graph consistency.
- LLMForEM (general-purpose entity resolution): This method uses an LLM to determine whether two entity records refer to the same entity. We retain the official zero-shot matching prompts and attribute–value serialization, mapping each mathematical node's type, title, and statement to an input record. It evaluates 20 candidates of the same type, returns the highest-ranked positive match, and returns NIL if all judgments are negative.
All three baselines retain their respective core fusion or matching mechanisms without using Gangmu's SEV or multi-evidence verification procedure. The comparisons use GPT-5.6-terra as the generation or judgment model throughout, while preserving each method's original retrieval and decision design. The MathER-Alg adaptations and experimental settings are detailed in Section 4.2 and Appendix C of the paper.
2. MathER-Alg: comparing existing fusion and entity matching methods.
We compare Gangmu, AutoMathKG, and LLMForEM on the same frozen KB and 512 expert-annotated queries. MathER-Alg requires linking a given mathematical statement to a strictly equivalent canonical node in the KB, or returning NIL when no match exists. KGGen constructs graphs through text extraction, aggregation, and deduplication; its official workflow does not provide a standalone query resolution interface for this task. We therefore exclude KGGen from the MathER-Alg experiment and compare it only in the end-to-end knowledge graph construction experiment below.
All three methods in this experiment use the same gold labels and evaluation metrics. The results are shown below; P and R denote precision and recall, respectively.
| Method | Linking P | NIL P | Linking R | NIL R | Overall Accuracy |
|---|---|---|---|---|---|
| AutoMathKG | 59.31% | 91.28% | 80.00% | 64.26% | 71.48% |
| LLMForEM | 47.75% | 100.00% | 81.28% | 40.43% | 59.18% |
| Gangmu | 87.10% | 85.76% | 80.43% | 91.34% | 86.33% |
Gangmu exceeds AutoMathKG and LLMForEM in overall accuracy by 14.84 and 27.15 percentage points, respectively, while also achieving higher linking precision and NIL recall. These results show that semantic normalization, multi-evidence verification, and equivalence decision mechanisms designed for mathematical statements improve both strict-equivalence linking and identification of knowledge absent from the KB.
3. End-to-end knowledge graph construction: final graph quality and knowledge coverage.
We further run the complete graph construction pipelines of Gangmu, AutoMathKG, and KGGen on the same mathematical texts, from knowledge extraction through cross-document consolidation. We compare the nodes, edges, and merging results in the final graphs, as well as their coverage of the mathematical knowledge in the source texts.
The experiment uses group theory chapters from Judson's Abstract Algebra: Theory and Applications, Goodman's Algebra: Abstract and Concrete, and Milne's Group Theory, totaling 69 subsections. All three methods read the same OCR text and use the same generation model while executing their respective extraction and cross-document consolidation procedures. Separately, model-assisted annotation and contextual review of these source texts identify 874 units of mathematical knowledge for assessing whether the final graphs fully preserve the source knowledge. These annotations are used only for evaluation and are not provided to the graph construction methods.
We use GPT-6.1-sol (high) to evaluate four metrics: node correctness checks whether the knowledge expressed by final graph nodes is supported by the source text; edge correctness checks the relations between nodes and their direction; merge correctness checks whether different formulations merged into the same node are indeed equivalent; and knowledge coverage measures the fraction of the 874 knowledge units fully expressed by the final graph. For Gangmu, the evaluated nodes and edges are the canonical nodes and edges in its final Canonical Graph. For the other methods, we evaluate the nodes and edges of their respective final graphs.
We sample up to 300 nodes and 300 edges per method, evaluating all items when fewer than 300 are available. Merge correctness is assessed through pairwise equivalence of pre-merge nodes assigned to the same final node, covering 85, 65, and 623 pairs for Gangmu, AutoMathKG, and KGGen, respectively.
W2 (continued)
Knowledge coverage requires preservation of mathematical objects, assumptions, quantifiers, scope, and conclusions. Multiple nodes, attributes, or triples may jointly express one knowledge unit, allowing different graph representations to be compared under the same criterion of complete mathematical content.
| Method | Node correctness | Edge correctness | Merge correctness | Knowledge coverage |
|---|---|---|---|---|
| AutoMathKG | 86.38% (184/213) | 90.00% (45/50) | 58.46% (38/65) | 50.46% (441/874) |
| KGGen | 91.33% (274/300) | 78.67% (236/300) | 42.86% (267/623) | 42.11% (368/874) |
| Gangmu | 93.67% (281/300) | 96.33% (289/300) | 76.47% (65/85) | 62.81% (549/874) |
Correctness rates in the table are the number of confirmed correct items divided by the total number of evaluated samples; knowledge coverage is the number of knowledge units confirmed to be fully expressed divided by 874. KGGen's node and merge scores are based on reevaluation with additional source context, while its edge and knowledge coverage scores retain the original evaluation results. Among its merge evaluations, 267 pairs are confirmed correct, 200 are incorrect, 66 are uncertain, and 90 encounter execution failures. The last two categories remain unknown and are not treated as mathematical errors. Gangmu's nodes express complete mathematical statements, whereas KGGen's nodes primarily express entities or concepts; knowledge coverage consistently assesses whether each graph fully expresses the mathematical content of the source text.
Gangmu achieves 62.81% knowledge coverage, exceeding AutoMathKG and KGGen by 12.36 and 20.71 percentage points, respectively, while also achieving high edge correctness. These results show that Gangmu preserves more mathematical content while constructing accurate knowledge relations. Together with the MathER-Alg results, the two experiments demonstrate its advantages in mathematical entity resolution and knowledge consolidation across documents.
W3: Mathematics-specific challenges and the need for specialized design
Thank you for raising this important question. Compared with common general-purpose knowledge graphs organized around entities, mathematical knowledge consolidation differs in four main respects.
The units of consolidation differ. General-purpose graphs typically consolidate names, attributes, and relations describing the same object. Mathematical knowledge consolidation handles complete definitions and theorems, requiring preservation of “which conclusions hold under which conditions,” beyond identifying the concepts involved.
The merging criteria differ. General-purpose entity resolution primarily determines whether records refer to the same object. Merging mathematical statements requires strict mathematical semantic equivalence. Discussing the same object, having similar formulas, or even exhibiting a one-way implication is insufficient to justify merging. General theorems, special cases, and stronger or weaker conclusions must remain distinguishable.
Context plays a more decisive role in determining meaning. Mathematical symbols, domains, and local conventions are often specified earlier in the text and directly determine a proposition's meaning and scope. Comparing two passages therefore often requires recovering the definitions they depend on. Different notation may express the same knowledge, while similar formulas may express different propositions.
Strict mathematical semantic equivalence is required. General-purpose entity resolution checks whether records refer to the same entity. Merging mathematical statements requires strict equivalence of their complete mathematical content, including domains, assumptions, quantifiers, conclusions, and logical direction. Omitting a condition or changing a quantifier can alter the proposition; high similarity, partial overlap, or one-way implication cannot substitute for strict equivalence.
The following two examples from MathER-Alg illustrate these differences.
Example 1: Group center — different notation, strictly equivalent mathematical meaning
Statement A (query)
Let be a group and define
The set is called the center of .
Statement B (canonical statement in the KB)
For a group , its center is the set
Equivalence analysis: strictly equivalent. Renaming to and to in A, then exchanging the two sides of the equality, yields the condition in B. The domains, universal quantifiers, and commutativity conditions are identical, defining the same subset .
Gangmu's decision: confirmed equivalence. The query is linked to canonical node 200, matching the human annotation.
Response to Reviewer jKxW
Response to Reviewer jKxW
Thank you for your valuable feedback. We have added relevant experiments and clarifications and respond to each point below.
W1: Human evaluation of the end-to-end Canonical Graph
Thank you for highlighting the importance of directly evaluating the final Canonical Graph. We added a human evaluation of samples from the final Canonical Graph to directly assess the output of Gangmu after knowledge extraction, semantic normalization, cross-document merging, and relation consolidation. The results show good performance in constructing canonical entities and relations, consistent with the stage-wise experiments. This experiment has been added to Section 4.4 of the paper.
The experiment uses the 30 textbooks used to construct the MathER-Alg reference KB. We evaluate the automatically generated Canonical Graph before manual correction, which contains 8,876 canonical nodes and 27,454 canonical edges. We randomly sample 100 canonical nodes with multiple evidence nodes, 100 canonical nodes with a single evidence node, and 100 canonical edges. The edge sample includes 50 edges supported by a single evidence edge and 50 supported by multiple evidence edges. An expert reviews each sample, and uncertain cases are resolved through discussion to determine the final labels.
The evaluation uses criteria specific to each of the three categories:
A sample is counted as correct only if it passes all applicable criteria for its category. The results are as follows.
These results directly demonstrate the quality of Gangmu's end-to-end output: mathematical statements from different sources are effectively consolidated into canonical entities, with good correctness of the normalized knowledge and its relations. Together with the Evidence Graph evaluation and the MathER experiments, this assessment covers local knowledge extraction, cross-document entity resolution, and the quality of the final Canonical Graph. The supplementary directory
Experiments/Canonical_Graph_Correctness_Evaluation/provides the fixed samples, evaluation content, and final labels.W2: Comparisons with existing graph construction and entity resolution methods
Thank you for raising this issue. We agree that the original experiments provided insufficient comparisons with existing methods for mathematical knowledge graph construction, general-purpose graph construction, and entity resolution. We therefore introduced AutoMathKG, KGGen, and LLMForEM in two experiments: entity resolution on MathER-Alg and end-to-end knowledge graph construction. These experiments directly compare the methods' performance in mathematical semantic matching and knowledge consolidation.
1. Method selection and adaptation.