acceptodds
Under review as a conference paper at ICLR 2027

Gangmu: Quotient-Based Consolidation of Mathematical Knowledge Across Documents

Abstract

Mathematical statements often appear in different forms across documents, yet systematically identifying and consolidating equivalent statements remains underexplored. We formulate Mathematical Knowledge Consolidation (MKC) using the Quotient-Based Graph Structure, which maps document-level statement nodes to canonical mathematical entities under strict semantic equivalence while preserving source evidence. Building on this structure, we introduce Gangmu, a two-stage framework that constructs an Evidence Graph of statements and relations and recovers a Canonical Graph through Mathematical Entity Resolution (MathER) in an open-world setting that permits unmatched statements. Gangmu uses semantic normalization, multi-evidence representations, uncertainty handling, and consistency checks to restrict linking and merging to confirmed equivalence. We also introduce MathER-Alg, a benchmark built from 50 algebra texts, comprising a reference knowledge base and 512 expert-annotated queries for open-world mathematical entity resolution. Experiments on MathER-Alg show that Gangmu achieves the highest overall accuracy among the evaluated methods.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
AI for Math, Mathematical Knowledge Graphs, Large Language Models, Mathematical Entity Resolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jKxWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on Mathematical Knowledge Consolidation (MKC), which aims to construct consolidated mathematical knowledge graphs from mathematical documents. Specifically, it proposes Gangmu, a two-stage framework that first constructs an Evidence Graph and then canonicalizes mathematical statements through entity resolution. The paper also introduces MathER-Alg, a benchmark for mathematical entity resolution. The experimental results demonstrate the effectiveness of the proposed method on the benchmark.

The paper proposes a practical framework for mathematical knowledge consolidation and introduces a potentially useful benchmark for mathematical entity resolution. However, the experimental evaluation is insufficient to establish the superiority of the proposed framework over existing methods, particularly for the complete MKC task. Overall, this reviewer leans toward a negative score.

Critical strengths
  1. The two-stage framework, which separates mathematical knowledge extraction from entity resolution and canonicalization, is practical and reasonably designed.
  2. The introduced mathematical entity resolution benchmark may serve as a useful resource for future research.
  3. The methodology and experimental settings are described in sufficient detail.
Critical weaknesses
  1. The paper evaluates the two stages separately, namely Evidence Graph construction and mathematical entity resolution. However, the effectiveness of the complete MKC framework remains unclear, as the quality of the final Canonical Graph is not directly evaluated.
  2. The comparisons with existing methods are insufficient. First, the paper does not compare Gangmu with existing mathematical knowledge graph construction methods or general-purpose knowledge graph construction and consolidation methods adapted to mathematical documents. Second, the MathER evaluation mainly includes retrieval-based baselines and a general-purpose agent, without comparisons against established entity resolution methods. These missing comparisons make it difficult to assess the advantages of the proposed framework over existing techniques.
  3. The discussion of mathematics-specific challenges is insufficient. It remains unclear what makes these challenges fundamentally different in the mathematical domain and why existing general-purpose methods would be inadequate.

Reference:

[1] Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to Reviewer jKxW

Response to Reviewer jKxW

Thank you for your valuable feedback. We have added relevant experiments and clarifications and respond to each point below.

W1: Human evaluation of the end-to-end Canonical Graph

Thank you for highlighting the importance of directly evaluating the final Canonical Graph. We added a human evaluation of samples from the final Canonical Graph to directly assess the output of Gangmu after knowledge extraction, semantic normalization, cross-document merging, and relation consolidation. The results show good performance in constructing canonical entities and relations, consistent with the stage-wise experiments. This experiment has been added to Section 4.4 of the paper.

The experiment uses the 30 textbooks used to construct the MathER-Alg reference KB. We evaluate the automatically generated Canonical Graph before manual correction, which contains 8,876 canonical nodes and 27,454 canonical edges. We randomly sample 100 canonical nodes with multiple evidence nodes, 100 canonical nodes with a single evidence node, and 100 canonical edges. The edge sample includes 50 edges supported by a single evidence edge and 50 supported by multiple evidence edges. An expert reviews each sample, and uncertain cases are resolved through discussion to determine the final labels.

The evaluation uses criteria specific to each of the three categories:

  1. Canonical nodes with multiple evidence nodes: merge correctness and consistency of the canonical statement. We check pairwise mathematical equivalence among the canonical representative statement and the normalized statements of all members, examining mathematical objects, scope, assumptions, quantifiers, conclusions, and logical direction.
  2. Canonical nodes with a single evidence node: content correctness and semantic fidelity. We check whether metadata such as title, type, and domain match the content, whether the original statement is correct under the textbook's conventions, and whether the normalized statement faithfully preserves its mathematical meaning. In this sample, each canonical statement is identical to the normalized statement of its sole member.
  3. Canonical edges: relation correctness. We check the relation between the canonical endpoints, its direction, and every recorded relation type. Statement dependencies require explicit support from the mathematical content. Proof relations require verification that the source knowledge is actually used in the supplied proof of the target statement. Edges recording both relation types must satisfy both criteria.

A sample is counted as correct only if it passes all applicable criteria for its category. The results are as follows.

Evaluation targetSample sizeCorrectHuman evaluation pass rate
Canonical nodes with multiple evidence nodes1008787.0%
Canonical nodes with a single evidence node1009191.0%
Canonical edges1008787.0%

These results directly demonstrate the quality of Gangmu's end-to-end output: mathematical statements from different sources are effectively consolidated into canonical entities, with good correctness of the normalized knowledge and its relations. Together with the Evidence Graph evaluation and the MathER experiments, this assessment covers local knowledge extraction, cross-document entity resolution, and the quality of the final Canonical Graph. The supplementary directory Experiments/Canonical_Graph_Correctness_Evaluation/ provides the fixed samples, evaluation content, and final labels.

W2: Comparisons with existing graph construction and entity resolution methods

Thank you for raising this issue. We agree that the original experiments provided insufficient comparisons with existing methods for mathematical knowledge graph construction, general-purpose graph construction, and entity resolution. We therefore introduced AutoMathKG, KGGen, and LLMForEM in two experiments: entity resolution on MathER-Alg and end-to-end knowledge graph construction. These experiments directly compare the methods' performance in mathematical semantic matching and knowledge consolidation.

1. Method selection and adaptation.

  • AutoMathKG (mathematical knowledge graph construction): This method constructs mathematical knowledge graphs through entity and relation extraction, attribute enrichment, and entity fusion. We reproduce its procedure from the published paper, retaining the MathVD1 representation, MiniLM encoder, top-5 retrieval, pairwise equivalence judgments, and selection rule for multiple matches. For MathER-Alg, we use its entity fusion module and map decisions to create a new entity to NIL. For the graph construction experiment, we convert explicit mathematical declarations in the shared OCR Markdown into the format required by its rule-based extractor, preserving the original content without adding content based on the evaluation answers.

W2 (continued)

  • KGGen (general-purpose knowledge graph construction): This method extracts entity–relation triples from text and consolidates the graph through aggregation, embedding-based clustering, and LLM-based deduplication. Using the official code, we retain these core steps and the original graph representation, making the necessary adaptations and fixes to the model interface, provenance recording, and graph consistency.
  • LLMForEM (general-purpose entity resolution): This method uses an LLM to determine whether two entity records refer to the same entity. We retain the official zero-shot matching prompts and attribute–value serialization, mapping each mathematical node's type, title, and statement to an input record. It evaluates 20 candidates of the same type, returns the highest-ranked positive match, and returns NIL if all judgments are negative.

All three baselines retain their respective core fusion or matching mechanisms without using Gangmu's SEV or multi-evidence verification procedure. The comparisons use GPT-5.6-terra as the generation or judgment model throughout, while preserving each method's original retrieval and decision design. The MathER-Alg adaptations and experimental settings are detailed in Section 4.2 and Appendix C of the paper.

2. MathER-Alg: comparing existing fusion and entity matching methods.

We compare Gangmu, AutoMathKG, and LLMForEM on the same frozen KB and 512 expert-annotated queries. MathER-Alg requires linking a given mathematical statement to a strictly equivalent canonical node in the KB, or returning NIL when no match exists. KGGen constructs graphs through text extraction, aggregation, and deduplication; its official workflow does not provide a standalone query resolution interface for this task. We therefore exclude KGGen from the MathER-Alg experiment and compare it only in the end-to-end knowledge graph construction experiment below.

All three methods in this experiment use the same gold labels and evaluation metrics. The results are shown below; P and R denote precision and recall, respectively.

MethodLinking PNIL PLinking RNIL ROverall Accuracy
AutoMathKG59.31%91.28%80.00%64.26%71.48%
LLMForEM47.75%100.00%81.28%40.43%59.18%
Gangmu87.10%85.76%80.43%91.34%86.33%

Gangmu exceeds AutoMathKG and LLMForEM in overall accuracy by 14.84 and 27.15 percentage points, respectively, while also achieving higher linking precision and NIL recall. These results show that semantic normalization, multi-evidence verification, and equivalence decision mechanisms designed for mathematical statements improve both strict-equivalence linking and identification of knowledge absent from the KB.

3. End-to-end knowledge graph construction: final graph quality and knowledge coverage.

We further run the complete graph construction pipelines of Gangmu, AutoMathKG, and KGGen on the same mathematical texts, from knowledge extraction through cross-document consolidation. We compare the nodes, edges, and merging results in the final graphs, as well as their coverage of the mathematical knowledge in the source texts.

The experiment uses group theory chapters from Judson's Abstract Algebra: Theory and Applications, Goodman's Algebra: Abstract and Concrete, and Milne's Group Theory, totaling 69 subsections. All three methods read the same OCR text and use the same generation model while executing their respective extraction and cross-document consolidation procedures. Separately, model-assisted annotation and contextual review of these source texts identify 874 units of mathematical knowledge for assessing whether the final graphs fully preserve the source knowledge. These annotations are used only for evaluation and are not provided to the graph construction methods.

We use GPT-6.1-sol (high) to evaluate four metrics: node correctness checks whether the knowledge expressed by final graph nodes is supported by the source text; edge correctness checks the relations between nodes and their direction; merge correctness checks whether different formulations merged into the same node are indeed equivalent; and knowledge coverage measures the fraction of the 874 knowledge units fully expressed by the final graph. For Gangmu, the evaluated nodes and edges are the canonical nodes and edges in its final Canonical Graph. For the other methods, we evaluate the nodes and edges of their respective final graphs.

We sample up to 300 nodes and 300 edges per method, evaluating all items when fewer than 300 are available. Merge correctness is assessed through pairwise equivalence of pre-merge nodes assigned to the same final node, covering 85, 65, and 623 pairs for Gangmu, AutoMathKG, and KGGen, respectively.

W2 (continued)

Knowledge coverage requires preservation of mathematical objects, assumptions, quantifiers, scope, and conclusions. Multiple nodes, attributes, or triples may jointly express one knowledge unit, allowing different graph representations to be compared under the same criterion of complete mathematical content.

MethodNode correctnessEdge correctnessMerge correctnessKnowledge coverage
AutoMathKG86.38% (184/213)90.00% (45/50)58.46% (38/65)50.46% (441/874)
KGGen91.33% (274/300)78.67% (236/300)42.86% (267/623)42.11% (368/874)
Gangmu93.67% (281/300)96.33% (289/300)76.47% (65/85)62.81% (549/874)

Correctness rates in the table are the number of confirmed correct items divided by the total number of evaluated samples; knowledge coverage is the number of knowledge units confirmed to be fully expressed divided by 874. KGGen's node and merge scores are based on reevaluation with additional source context, while its edge and knowledge coverage scores retain the original evaluation results. Among its merge evaluations, 267 pairs are confirmed correct, 200 are incorrect, 66 are uncertain, and 90 encounter execution failures. The last two categories remain unknown and are not treated as mathematical errors. Gangmu's nodes express complete mathematical statements, whereas KGGen's nodes primarily express entities or concepts; knowledge coverage consistently assesses whether each graph fully expresses the mathematical content of the source text.

Gangmu achieves 62.81% knowledge coverage, exceeding AutoMathKG and KGGen by 12.36 and 20.71 percentage points, respectively, while also achieving high edge correctness. These results show that Gangmu preserves more mathematical content while constructing accurate knowledge relations. Together with the MathER-Alg results, the two experiments demonstrate its advantages in mathematical entity resolution and knowledge consolidation across documents.

W3: Mathematics-specific challenges and the need for specialized design

Thank you for raising this important question. Compared with common general-purpose knowledge graphs organized around entities, mathematical knowledge consolidation differs in four main respects.

The units of consolidation differ. General-purpose graphs typically consolidate names, attributes, and relations describing the same object. Mathematical knowledge consolidation handles complete definitions and theorems, requiring preservation of “which conclusions hold under which conditions,” beyond identifying the concepts involved.

The merging criteria differ. General-purpose entity resolution primarily determines whether records refer to the same object. Merging mathematical statements requires strict mathematical semantic equivalence. Discussing the same object, having similar formulas, or even exhibiting a one-way implication is insufficient to justify merging. General theorems, special cases, and stronger or weaker conclusions must remain distinguishable.

Context plays a more decisive role in determining meaning. Mathematical symbols, domains, and local conventions are often specified earlier in the text and directly determine a proposition's meaning and scope. Comparing two passages therefore often requires recovering the definitions they depend on. Different notation may express the same knowledge, while similar formulas may express different propositions.

Strict mathematical semantic equivalence is required. General-purpose entity resolution checks whether records refer to the same entity. Merging mathematical statements requires strict equivalence of their complete mathematical content, including domains, assumptions, quantifiers, conclusions, and logical direction. Omitting a condition or changing a quantifier can alter the proposition; high similarity, partial overlap, or one-way implication cannot substitute for strict equivalence.

The following two examples from MathER-Alg illustrate these differences.

Example 1: Group center — different notation, strictly equivalent mathematical meaning

Statement A (query)

Let GG be a group and define

Z(G)={z∈G:∀a∈G,  az=za}.Z(G)=\{z\in G:\forall a\in G,\;az=za\}.

The set Z(G)Z(G) is called the center of GG.

Statement B (canonical statement in the KB)

For a group GG, its center is the set

Z(G)={x∈G:∀g∈G,  xg=gx}.Z(G)=\{x\in G:\forall g\in G,\;xg=gx\}.

Equivalence analysis: strictly equivalent. Renaming zz to xx and aa to gg in A, then exchanging the two sides of the equality, yields the condition in B. The domains, universal quantifiers, and commutativity conditions are identical, defining the same subset Z(G)Z(G).

Gangmu's decision: confirmed equivalence. The query is linked to canonical node 200, matching the human annotation.

Reviewer_Pac9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper defines Mathematical Knowledge Consolidation (MKC), a task that aims to identify definitions and theorems with strictly equivalent meanings across mathematical documents, consolidate them into shared canonical entities, and preserve their original statements, sources, and document-level relations.

The paper contributes both a method and a benchmark for this task. The proposed method, Gangmu, first parses mathematical documents and extracts statement nodes and their dependencies to construct the Evidence Graph. It then recovers the Canonical Graph by retrieving candidate entities and conservatively merging only those confirmed to be semantically equivalent. The proposed benchmark, MathER-Alg, is constructed from 50 algebra texts. Thirty documents are used to build a reference knowledge base, while the remaining twenty documents provide 512 expert-curated queries.

On the frozen-reference-KB entity-resolution task, Gangmu achieves 86.33% overall accuracy, compared with 80.86% for the strongest reported baseline. Four ablation studies suggest that semantic normalization, multi-evidence candidate representation, reranking, and explicit uncertainty handling all contribute to the reported entity-resolution performance.

Critical strengths

The paper defines a meaningful problem by explicitly distinguishing mathematical relatedness from strict semantic equivalence, an important distinction for reliably consolidating mathematical knowledge across documents.

The paper documents many steps of benchmark construction and acknowledges licensing and evaluation limitations more openly than many submissions.

Critical weaknesses

The evaluation does not match the end-to-end claim. All main resolution experiments use a generated and frozen reference KB, and the reported metrics (Overall Accuracy, Linking Precision/Recall, and NIL Precision/Recall) only evaluate query-to-KB resolution.

Baselines are weak, partially non-independent, and insufficiently specified.

LLMs are used throughout benchmark construction, inference, and evaluation without sufficient stage-wise human validation. The reported expert comparison covers only judge agreement on 100 sampled nodes and 100 sampled edges.

Questions for authors

Can the authors provide an end-to-end evaluation on a fully annotated multi-document subset?

Can the authors evaluate whether the recovered graph improves a downstream task such as theorem retrieval, premise selection, mathematical question answering, or proof construction?

Can an independently runnable version be provided without access to restricted source texts?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Pac9

Response to Reviewer Pac9

Thank you for your valuable feedback. We have added relevant experiments and clarifications and respond to each point below.

W1 & Q1: Human evaluation of the end-to-end Canonical Graph

Thank you for highlighting the importance of directly evaluating the results of end-to-end consolidation. We added a human evaluation of samples from the final Canonical Graph to directly assess the output of Gangmu after knowledge extraction, semantic normalization, cross-document merging, and relation consolidation. The results show good performance in constructing canonical entities and relations, consistent with the stage-wise experiments. This experiment has been added to Section 4.4 of the paper.

The experiment uses the 30 textbooks used to construct the MathER-Alg reference KB. We evaluate the automatically generated Canonical Graph before manual correction, which contains 8,876 canonical nodes and 27,454 canonical edges. We randomly sample 100 canonical nodes with multiple evidence nodes, 100 canonical nodes with a single evidence node, and 100 canonical edges. The edge sample includes 50 edges supported by a single evidence edge and 50 supported by multiple evidence edges. An expert reviews each sample, and uncertain cases are resolved through discussion to determine the final labels.

The evaluation uses criteria specific to each of the three categories:

  1. Canonical nodes with multiple evidence nodes: merge correctness and consistency of the canonical statement. We check pairwise mathematical equivalence among the canonical representative statement and the normalized statements of all members, examining mathematical objects, scope, assumptions, quantifiers, conclusions, and logical direction.
  2. Canonical nodes with a single evidence node: content correctness and semantic fidelity. We check whether metadata such as title, type, and domain match the content, whether the original statement is correct under the textbook's conventions, and whether the normalized statement faithfully preserves its mathematical meaning. In this sample, each canonical statement is identical to the normalized statement of its sole member.
  3. Canonical edges: relation correctness. We check the relation between the canonical endpoints, its direction, and every recorded relation type. Statement dependencies require explicit support from the mathematical content. Proof relations require verification that the source knowledge is actually used in the supplied proof of the target statement. Edges recording both relation types must satisfy both criteria.

A sample is counted as correct only if it passes all applicable criteria for its category. The results are as follows.

Evaluation targetSample sizeCorrectHuman evaluation pass rate
Canonical nodes with multiple evidence nodes1008787.0%
Canonical nodes with a single evidence node1009191.0%
Canonical edges1008787.0%

These results directly demonstrate the quality of Gangmu's end-to-end output: mathematical statements from different sources are effectively consolidated into canonical entities, with good correctness of the normalized knowledge and its relations. Together with the Evidence Graph evaluation and the MathER experiments, this assessment covers local knowledge extraction, cross-document entity resolution, and the quality of the final Canonical Graph. The supplementary directory Experiments/Canonical_Graph_Correctness_Evaluation/ provides the fixed samples, evaluation content, and final labels.

W2: Baseline competitiveness, independence, and implementation details

Thank you for this suggestion. Building on the existing descriptions in Section 4.2 and Appendix C of the paper, we have provided further details on the implementation and experimental settings of each baseline, including input representations, candidate retrieval, model configurations, decision rules, and shared components. These details make the comparisons clearer and easier to reproduce.

We also agree that stronger baselines were needed and that some of the original methods share components with Gangmu, leaving insufficient comparison with independent methods. We therefore added AutoMathKG and LLMForEM as independent baselines based on established methods, covering mathematical knowledge graph construction and general-purpose entity resolution, respectively:

  • AutoMathKG (mathematical knowledge graph construction): This method uses LLMs to extract mathematical entities and relations from mathematical documents, then applies entity fusion to consolidate different formulations of the same mathematical knowledge. We adapt its entity fusion procedure to MathER-Alg, retaining the MathVD1 representation, original MiniLM encoder, top-5 retrieval, pairwise equivalence judgments, and selection rule for multiple matches as described in the published paper. Decisions to create a new entity are mapped to NIL.
  • LLMForEM (general-purpose entity resolution): This method converts entity records into textual descriptions and uses an LLM to determine whether two records refer to the same entity. We use its official zero-shot matching setup, retaining the original prompts and attribute–value serialization. It evaluates 20 candidates of the same type obtained through dense retrieval, returns the highest-ranked positive match, and returns NIL if all judgments are negative.

The core fusion and matching decisions of both baselines are fully independent of Gangmu. Each uses its own original judgment prompts and decision rules, without Gangmu's SEV or multi-evidence verification mechanisms. To align the comparison conditions, Gangmu and the new baselines use the same GPT-5.6-terra backbone, frozen KB, and 512 queries while retaining their respective candidate generation and matching rules. The results are shown below; P and R denote precision and recall, respectively.

MethodLinking PNIL PLinking RNIL ROverall Accuracy
AutoMathKG59.31%91.28%80.00%64.26%71.48%
LLMForEM47.75%100.00%81.28%40.43%59.18%
Gangmu87.10%85.76%80.43%91.34%86.33%

Gangmu exceeds AutoMathKG and LLMForEM in overall accuracy by 14.84 and 27.15 percentage points, respectively, while also achieving higher linking precision and NIL recall. These results demonstrate its advantages in confirming strictly equivalent matches and identifying knowledge absent from the KB.

Q3: Independent execution and reproduction without restricted source texts

Thank you for your attention to reproducibility and reusability. We provide a version of the MathER-Alg evaluation that runs without access to the original textbook PDFs. The supplementary materials contain the complete frozen KB, 512 queries with gold labels, the evidence-to-canonical mapping, and associated embeddings. The evaluation programs read these data directly.

The anonymous code repository provides the complete source code, prompts, and usage instructions. After installing the dependencies and configuring model services as described in the README, extract the supplementary materials into the repository's Material/ directory and run:

# Evaluate Gangmu and eight baselines on MathER-Alg
python mather_evaluation.py

# Run the four ablation experiments
python ablation_study.py

# Evaluate Evidence Graph quality on the provided fixed samples
python evidence_graph_evaluation.py

The benchmark data are located in MathER-Alg/Knowledge_Base/ and MathER-Alg/Query_Set/, while the fixed evaluation samples and related experimental results are in Experiments/. These experiments run directly on the supplementary data without downloading or parsing the original textbooks again.

The repository also provides a standalone entry point for the complete PDF-to-graph pipeline, allowing users to process their own mathematical documents:

python main.py --input-dir inputs --output-dir outputs/documents

In addition, MathER-Alg/textbooks.md in the supplementary materials lists textbook sources and download links for provenance tracing. The supplied benchmark data support evaluation reproduction, while the complete pipeline allows users to select and process their own documents, supporting both reproducibility and reuse of the framework.

W3: Human validation in benchmark construction, model inference, and evaluation

Thank you for your attention to validation at each stage. Gangmu uses LLMs at multiple stages, and we place strong emphasis on expert annotation, explicit mathematical criteria, and human evaluation of the final output to support result quality. We describe the measures taken at each stage below.

1. Benchmark construction: model-assisted sampling and independently determined expert gold labels.

Appendix B of the paper details the construction and annotation of MathER-Alg. LLMs assist with initial graph construction and query sampling; experts review potentially equivalent nodes in the KB and independently determine the query gold labels. Experts are not shown the model's preliminary judgments during annotation, and model outputs are not directly used as gold labels. All 512 released queries and their labels are confirmed by human annotators and frozen together with the KB for evaluation of all methods.

2. Model inference: multiple layers of verification guided by strict mathematical equivalence.

Current LLM judgments do not provide a formal guarantee of strict mathematical equivalence. To support this requirement, Gangmu first uses dependency context to clarify mathematical objects, symbols, domains, and local conventions while preserving the original assumptions, quantifiers, and conclusions during semantic normalization. It then combines center evidence, complementary representative evidence, and query-matched evidence to examine the full mathematical meaning of each candidate, explicitly distinguishing strict equivalence from generalizations, special cases, and one-way implications.

The decision stage uses three outcomes: “confirmed equivalence,” “potential equivalence,” and “non-equivalence.” Only confirmed equivalence can authorize linking, and unresolved cases remain separate. Merging multiple candidates also requires pairwise equivalence verification and consistency checks. After generating a canonical statement, Gangmu further checks the fidelity of its mathematical scope, assumptions, quantifiers, and conclusions, retaining source evidence and mappings for traceable verification. The ablation experiments in the paper validate the effectiveness of these mechanisms for mathematical entity resolution and linking across documents.

3. Evaluation: extending expert verification of the Evidence Graph to direct human evaluation of the final Canonical Graph.

For the Evidence Graph evaluation, a judge model independent of the construction model assesses 500 nodes and 500 edges. Experts then review 100 nodes and 100 edges from these samples. Agreement between the judge and experts is 96% for nodes and 95% for edges, indicating high agreement on the reviewed samples.

For the final consolidation results, we additionally conduct human evaluation of 100 canonical nodes with multiple evidence nodes, 100 canonical nodes with a single evidence node, and 100 canonical edges. The expert assessment examines mathematical equivalence after merging, semantic fidelity of statements, and canonical relations. Proof relations are checked against the target proof for actual use of the source knowledge, and uncertain cases are resolved through discussion. This evaluation directly assesses the final Canonical Graph; its procedure and results are provided in our response to W1 & Q1 above. Together, these efforts provide human validation of the benchmark gold labels, judge agreement, and final output quality.

Reviewer_zpb7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Gangmu uses semantic normalization and LLM-based verification to merge equivalent mathematical statements across documents into a unified knowledge graph while preserving their sources.

Critical strengths

Gangmu consolidates equivalent mathematical statements while preserving source evidence and avoiding the conflation of similar but mathematically distinct claims.

Critical weaknesses
  1. Gangmu relies on LLM-based judgments that cannot guarantee strict mathematical equivalence.
  2. What is the computational cost of consolidating a large document collection?
Questions for authors
  1. How well does the framework generalize beyond algebra?
  2. Does the resulting graph improve downstream retrieval or theorem proving?
  3. It seems that the MathER-Alg KB and annotations are not included in the codebase. Are they missing?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zpb7

Response to Reviewer zpb7

Thank you for your valuable feedback. We have added relevant experiments and clarifications and respond to each point below.

W1: LLM-based judgments cannot guarantee strict mathematical equivalence

Thank you for highlighting this important issue. Strict mathematical equivalence is central to mathematical knowledge consolidation. We agree that current LLM-based judgments cannot guarantee strict mathematical semantic equivalence. Section 3.3.1 of the paper explicitly distinguishes the ideal semantic equivalence relation from the node partition recovered in practice.

To ensure strict mathematical equivalence as far as possible, Gangmu uses multiple mechanisms across statement normalization, equivalence verification, and canonical content generation:

  1. Semantic normalization that preserves mathematical conditions. Relevant dependency context is used to clarify symbols, object domains, and local conventions while preserving the original assumptions, quantifiers, and conclusions. This allows comparisons across documents to account for the full mathematical meaning of each statement.
  2. Semantic verification using multiple evidence statements. Candidate entities are represented using center evidence, complementary representative evidence, and query-matched evidence when available. Semantic Equivalence Verification (SEV) checks domains, assumptions, quantifiers, conclusions, and definitional conventions, distinguishing strict equivalence from generalizations, special cases, and one-way implications.
  3. Three-way judgments and conservative merging. SEV distinguishes “confirmed equivalence,” “potential equivalence,” and “non-equivalence.” Only confirmed equivalence can authorize linking or merging. When multiple candidate entities are involved, they must also pass pairwise equivalence verification with no unresolved merge conflicts; nodes without confirmed matches remain separate.
  4. Verification of canonical statements and preservation of evidence. After generating a canonical statement, Gangmu checks for changes to mathematical scope, assumptions, quantifiers, and conclusions. It also retains the original evidence and its mapping to the Canonical Graph to support tracing and correction.

The ablation experiments in the paper further validate the effectiveness of semantic normalization, multi-evidence representations, and three-way equivalence verification, showing that these mechanisms improve mathematical entity resolution and linking across documents.

We are exploring the integration of proof assistants such as Lean 4 into Gangmu to combine semantic equivalence judgments with machine-checkable formal proofs and strengthen the assurance of mathematical equivalence. This requires faithful translation from natural language into a formal language, as well as additional formalization and proof-search costs, so this mechanism is not included in the present work. It remains an important direction for further developing Gangmu.

W2: Computational cost of consolidating a large document collection

Thank you for asking about computational cost. We measured usage from OCR through Canonical Graph generation using the execution logs for the 30 textbooks, totaling 7,917 pages, used to construct the MathER-Alg reference KB. The table below reports costs by stage and for the complete pipeline, estimated at official rates. Canonical Graph recovery includes semantic normalization.

StageGPT input (M tokens/book)GPT output (M tokens/book)Average cost (USD/book)Total cost for 30 books (USD)
OCR——2.0762.07
Evidence Graph construction2.21280.369411.07332.19
Canonical Graph recovery9.03780.314527.31819.37
End-to-end total (official rates)11.25060.683940.451,213.63

The official input/output rates are USD 2.50/15.00 per million tokens for GPT-5.4 and CNY 5/22 per million tokens for the OCR model GLM-5V-Turbo, as listed in the official OpenAI and Zhipu documentation. Costs in CNY are converted at USD 1 = CNY 7.

The table uses official API rates as a common reference. In practice, subscription plans such as ChatGPT Pro and the associated billing arrangements reduced our model usage costs, so the actual expenditure for this experiment was lower than the estimates in the table.

Q1: Cross-domain generalization beyond algebra

Thank you for raising this question. We added an experiment in mathematical analysis, and the results provide initial evidence that Gangmu generalizes well across mathematical domains.

The experiment uses four analysis textbooks: two on calculus and two on real analysis. We retain Gangmu's extraction, semantic normalization, retrieval, equivalence verification, and Canonical Graph construction procedures and prompts, changing only the default domain metadata from Algebra to Analysis. This run uses GPT-5.6-terra as the graph construction model and produces 880 evidence nodes, 753 canonical nodes, and 2,326 canonical edges.

We evaluate the final Canonical Graph using GPT-6.1-sol (high), sampling 100 canonical nodes with multiple evidence nodes, 100 canonical nodes with a single evidence node, and 100 canonical edges. Each sample receives a binary judgment under explicit criteria. For canonical nodes with multiple evidence nodes, the main criterion is equivalence between the canonical statement and the normalized statements of all members. For canonical nodes with a single evidence node, we assess content, metadata, and semantic fidelity from the original statement to its normalized version. For canonical edges, we check the relation, its direction, and all recorded relation types, providing the target proof as context when assessing proof relations.

Evaluation targetSample sizeJudged correctModel-assessed correctness
Canonical nodes with multiple evidence nodes1008181.0%
Canonical nodes with a single evidence node1009898.0%
Canonical edges1009292.0%

The results show that Gangmu can consolidate mathematical knowledge across documents in analysis, with good performance in semantic merging, statement fidelity, and relation construction. They demonstrate that Gangmu's design transfers from algebra to calculus and real analysis, supporting its cross-domain generalization.

Q3: Availability of the MathER-Alg KB and annotations

Thank you for your attention to data availability and reproducibility. The supplementary materials contain the complete frozen MathER-Alg benchmark, including the two-layer KB and 512 queries with gold labels. The relevant files are located at:

  • MathER-Alg/Knowledge_Base/Evidence_Graph/: Evidence Graph nodes, edges, provenance information, and supporting data.
  • MathER-Alg/Knowledge_Base/Canonical_Graph/: The frozen Canonical Graph with 8,842 nodes and 27,412 edges, together with the evidence-to-canonical mapping and supporting data.
  • MathER-Alg/Query_Set/queries.json: The 512 queries with their gold_label and gold_canonical_node_id fields, specifying linking targets or NIL. The same directory also provides provenance information and query embeddings.

The anonymous code repository provides the Gangmu framework and experiment code, prompts, and instructions for use and reproduction.

Q2: Does the Canonical Graph improve downstream retrieval or theorem proving?

Thank you for asking about downstream utility. We added a mathematical knowledge retrieval experiment covering definitions and theorems. The results show that, under identical retrieval scores, canonical consolidation across documents improves the coverage and ranking of relevant knowledge within a fixed result budget. We describe the setup and results below and have also added this experiment to Appendix F of the paper.

The experiment uses the graphs generated before manual correction from the 30 textbooks used to construct the MathER-Alg reference KB, comprising 9,935 evidence nodes and 8,876 canonical nodes. Starting from 100 canonical nodes (62 definitions and 38 theorems), we construct 127 English queries covering applications, proof goals, and conditions defining concepts. Domain experts review and refine the queries.

Relevance is annotated on three levels according to the knowledge's actual contribution to the original query:

  • 2: Directly relevant. When its applicability conditions are satisfied, the knowledge directly answers the query or establishes a specific key step needed to solve the problem.
  • 1: Supporting relevance. The knowledge supports a specific, substantive auxiliary step in solving the problem, such as supplying a required definition or supporting an intermediate derivation.
  • 0: Irrelevant. The knowledge has no identifiable practical use for the query, is only topically similar, or has applicability conditions that are not satisfied.

Pooling the results from both retrieval conditions with candidates from supplementary searches yields 2,176 relevance annotations for query–canonical entity pairs. GPT-6.1-sol makes the initial judgments, and experts review items flagged as uncertain by the model. Both retrieval conditions share these annotations, and every evaluated retrieval result has a definite relevance label.

Both retrieval conditions use the same query and evidence embeddings from Qwen3-Embedding-8B and exactly the same cosine similarity scores, with no query rewriting or reranking during retrieval:

  • Evidence: Returns the highest-scoring nodes from the union of document-level evidence nodes. Different formulations of the same knowledge entity may occupy multiple result positions.
  • Canonical: Scores each canonical entity by the highest score among its member evidence nodes and returns distinct canonical entities, each occupying one result position.
MetricEvidence@5Canonical@5Evidence@10Canonical@10
Hit0.95280.96850.99210.9921
Strict-Hit0.87400.88980.93700.9449
MRR0.79490.80500.80110.8085
Pooled-Knowledge-Recall0.29190.33170.49150.5651
Knowledge-nDCG0.52030.57560.54550.6049
Knowledge-MAP0.24060.28320.36660.4457

All metrics are averaged over the 127 queries. Hit counts relevance grades 1 or 2 as a hit, whereas Strict-Hit requires grade 2. The last three knowledge-level metrics credit each entity only once while retaining the positions occupied by duplicate results. The denominator of Pooled-Knowledge-Recall is the number of annotated relevant entities in the candidate pool, so it measures recall within that pool.

At @10, canonical retrieval improves knowledge recall from 0.4915 to 0.5651, Knowledge-nDCG from 0.5455 to 0.6049, and Knowledge-MAP from 0.3666 to 0.4457, while Hit@10 remains unchanged. In evidence retrieval, repeated knowledge occupies an average of 1.46 additional positions among the top ten results. Canonical retrieval eliminates these duplicate positions through consolidation, allowing the same result budget to accommodate more distinct relevant knowledge.

These results demonstrate the practical value of cross-document consolidation for retrieval: as the number of sources grows, organizing different formulations of the same knowledge under a shared entity allows a limited set of retrieval results to cover more useful knowledge. Preserving sources and dependencies also provides a unified organizational basis for expanding mathematical knowledge bases and reusing knowledge.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.