acceptodds
Under review as a conference paper at ICLR 2027

TopoMath: A Concept Graph Topology-Guided Approach for LLM Mathematical Reasoning

Abstract

Although large language models (LLMs) have shown immense potential for mathematical problem solving, their intermediate derivations frequently suffer from premise drift, computational errors, and exploratory "dead ends". A fundamental limitation is that models navigate an entirely implicit conceptual space without structural guidance on mathematical concepts and their interrelations. In this paper, we propose TopoMath, an inference-time framework that establishes a relationship graph of mathematical concepts to guide and verify LLM reasoning. Specifically, TopoMath builds a directed acyclic graph (DAG) from Wikidata's mathematical taxonomy and projects each problem onto a compact scaffold containing grounded anchors, taxonomic paths, shared ancestors, and local neighbors. At inference time, the scaffold can serve as a global navigational prior for solution generation, while a dual-verification mechanism combining SymPy's symbolic execution with topological concept-membership checks detects computational flaws and conceptual drifts to trigger targeted revision. Evaluated on six benchmarks spanning standard to Olympiad-level mathematics across three backbone models, TopoMath consistently outperforms direct inference and state-of-the-art inference-time baselines, exhibiting significant improvements on PASS@1 and perfect reasoning rates (PRRs). Process-level graph analysis further reveals that TopoMath eliminates 81.2% of irrelevant reasoning nodes and shortens proof depth by 34.5%, indicating that topological concept priors effectively foster concise, robust, and structurally closed mathematical reasoning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Mathematical Reasoning, Large Language Models, Inference-Time Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mEHdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TopoMath, a mathematical concept graph-based framework for LLM mathematical reasoning. The method first constructs a graph of mathematical concept relationships using Wikidata, then extracts relevant concepts and their relationships for each problem to form a scaffold that guides reasoning. It also combines SymPy-based execution verification with concept-membership checks to revise potentially problematic reasoning results. The authors conduct experiments on six mathematical reasoning benchmarks and three backbone models. The results show that the method consistently improves final-answer accuracy and achieves some improvements in structural metrics for reasoning processes.

Critical strengths

The research idea demonstrates a certain degree of novelty. The authors construct a graph based on relationships among mathematical concepts to assist LLM reasoning. Relevant mathematical concept information is provided before reasoning generation, and concept checks and symbolic execution verification are used for targeted revision afterward, forming a relatively comprehensive reasoning and feedback mechanism. Compared with traditional chain-of-thought and reasoning search methods, this work offers a useful perspective from the organization of mathematical knowledge.

The experiments are relatively comprehensive and involve a considerable amount of work. The method is evaluated on six benchmarks and three backbone models, compared with several mainstream inference-time baselines, and further examined through ablation studies and reasoning-process analyses. The results are reasonably convincing.

Critical weaknesses

The contribution of the core innovation could be further validated. Although the ablation study in Table 3 shows that the scaffold brings some performance improvements, it also introduces additional mathematical concepts. In my view, the current ablation study does not clearly distinguish whether the improvements come from the additional mathematical concepts or the topological relationships among them. A comparison that retains the same concept information without providing topological relationships would be helpful.

The ablation study of the dual-verification mechanism could be further improved. The authors design two verification mechanisms: execution-based verification and topology-based concept-membership verification. The existing ablation study does demonstrate a positive effect of the combined mechanism, but the individual effects of the two verification methods are not separately analyzed. Separate ablation studies would help clarify their respective contributions.

Questions for authors

If the model is provided with mathematical concepts but without the topological relationships among them, could it achieve similar experimental results? Such a comparison would help further clarify the contribution of the topological structure itself.

Could the authors provide separate ablation results for SymPy-based execution verification and topology-based concept-membership verification to clarify the respective contributions of the two mechanisms?

3 comments
AuthorsOpenReview ↗

Contributions of Topological Relations and Dual Verification

Thanks for the two suggestions. We agree that the original experiments did not sufficiently distinguish the contributions of concept information from that of topological relations, nor did they separately examine the effects of the two types of verification feedback. We thus conducted additional comparisons on Omni-500, as reported below.

Since commercial model API services may be updated over time and outputs may vary across runs, we re-evaluated the TopoMath pipeline and all ablation variants, using the complete pipeline from this evaluation batch as the common reference. Consequently, the absolute scores slightly differ from those in the original paper. All ablation differences are re-calculated within this new batch, without mixing original and re-evaluated results. Delta denotes the absolute change in Pass@1 relative to the TopoMath pipeline in this batch.

MethodAblation settingPass@1Delta
TopoMathComplete pipeline0.610-
w/o Dual VerificationRemoves both verifiers0.596-0.014
w/o ScaffoldRemoves scaffold guidance0.584-0.026
w/o Topological RelationsRemoves relations; retains concepts0.588-0.022
w/o Topology VerificationRemoves concept-membership checks0.582-0.028
w/o Execution VerificationRemoves SymPy verification0.596-0.014

(1) Distinguishing Concept Information from Topological Relations

To address this question, we added the w/o Topological Relations variant. It retains the same concepts and their available definitions as the complete TopoMath pipeline but replaces the structured scaffold with an unordered concept list. Concept paths, hierarchies, and connections are no longer explicitly presented, while both verifiers are retained. This setting differs from w/o Scaffold, which provides no scaffold guidance in the reasoning prompt.

The complete TopoMath pipeline achieves a Pass@1 of 0.610, compared with 0.588 for the concept-list control, a difference of 2.2 percentage points. Thus, the structured scaffold achieves higher accuracy even when the supplied concept information is held constant. This provides evidence for the incremental value of explicit topological relations within the current pipeline.

(2) Separately Examining Execution Verification and Concept-Membership Verification

We also added two individual ablations. w/o Topology Verification retains the scaffold and SymPy execution verification, removing only concept-membership checks. Its Pass@1 decreases to 0.582, a reduction of 2.8 percentage points relative to the complete pipeline. w/o Execution Verification retains the scaffold and concept-membership checks, removing only SymPy verification. Its Pass@1 is 0.596, a reduction of 1.4 percentage points. Removing both verifiers also yields a Pass@1 of 0.596.

The complete configuration achieves higher accuracy than each ablation variant, supporting the retention of both types of verification feedback. However, the reductions are not simply additive, and performance does not increase monotonically with the number of verifiers. These results reflect the contribution of each module within the current pipeline, rather than independent, linearly additive gains. We also distinguish the scope of the two verifiers: SymPy checks executable computational claims, whereas concept-membership verification checks whether the concepts used in the reasoning belong to the problem-relevant concept set.

Thanks for the detailed response. After reading the rebuttal and other reviews, I believe my current score is reasonable.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for taking the time to review our rebuttal and additional results. We appreciate your feedback and evaluation of our work.

Reviewer_gVuxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TopoMath is an inference-time framework that uses a mathematical concept graph to guide and refine LLM reasoning. It constructs a DAG from Wikidata, retrieves problem-specific concept scaffolds, and combines symbolic execution with concept-membership checks to provide feedback for targeted revision. Evaluation covers six mathematical reasoning benchmarks and three backbone models. The framework improves answer accuracy over direct inference in all 18 backbone–benchmark settings. With DeepSeekV4-Flash, it achieves the highest macro-average performance among the evaluated methods, alongside improvements in the structural closure and compactness of generated solutions.

Critical strengths
  1. A concrete and useful integration of structured knowledge with reasoning refinement. The paper develops an end-to-end approach that connects concept grounding, adaptive graph expansion, scaffold construction, and verification-guided revision.
  2. Broad empirical evidence of system-level effectiveness. The evaluation spans standard and competition-level mathematics, with consistent accuracy improvements over direct inference across three backbones. On the default backbone, the macro-average gains of 13.2 percentage points in PASS@1 and 29.8 percentage points in the operationally defined PRR are substantial.
Critical weaknesses
  1. Limited validation of the underlying concept graph. The paper reports graph statistics and structural checks, but provides limited evidence concerning concept coverage and the semantic accuracy of retained relations, particularly those introduced through manual curation or heuristic connections.
Questions for authors
  1. Since constructing the global mathematical concept graph is a central component of TopoMath, could the authors present its construction as a unified, step-by-step pipeline or algorithm? Appendix B.1 describes individual operations, but an integrated account of the relevant inputs, outputs, and selection rules would substantially improve its reproducibility and usefulness to the community.
  2. Beyond the reported node/edge counts and checks for acyclicity and root reachability, what validation was performed to assess the global graph’s quality? For example, are there measurements of the semantic correctness of retained relations, particularly manually introduced and heuristically constructed connections?
  3. The reproducibility statement mentions releasing code and data; does this include the complete graph-construction pipeline, extraction snapshots, curation rules, and final graph?
  4. Could the authors report TopoMath’s inference cost relative to the evaluated baselines, like total model calls? Does TopoMath offer a favorable accuracy–cost trade-off, and how does this change when scaffolds are cached and reused?
1 comment
AuthorsOpenReview ↗

Graph Construction, Validation, Reproducibility, and Inference Cost

Thank you for the constructive feedback. Revisions are marked by blue font in the original manuscript for ease of presentation. Below, we respond to each of your questions separately.

1. Unified graph-construction pipeline. We have added "Offline Mathematical Concept Topology Construction" to Appendix B.1. Algorithm 1 integrates root-based relation extraction, concept-layer enrichment, manual seed integration, graph governance, structural validation, and retrieval-asset generation into a pipeline. It specifies the inputs, selection and filtering rules, edge-merging procedures, and outputs, complementing the accompanying implementation details.

2. Graph quality and semantic validation. Beyond acyclicity and root reachability, our validation includes relation-source restrictions, domain-relevance filtering, and manual review of curated connections. The final offline graph contains 2,410 Wikidata-derived edges: 1,613 subclass, 499 part-of, and 298 facet-of relations. These are extracted from existing records and have been manually checked. We retain QIDs and relation types and apply predefined direction-conversion, deduplication, and filtering rules. Entities without basic label information are filtered; a blocklist removes clearly irrelevant gateways and branches that consequently become unreachable.

All 14 retained manual seed connections were individually reviewed against the relevant Wikidata entries for endpoint concepts, conceptual association, and connection direction. This documents the checks performed on the curated additions. Quantitative coverage and graph-wide semantic-accuracy rates, including a separate rate for heuristic connections, are not reported; Wikidata provenance and structural validity alone do not establish semantic correctness.

3. Released reconstruction materials. Yes. The anonymized release includes the complete construction scripts, cached extraction snapshots, curation rules (including blocklists and seed connections), final DAG, and derived retrieval assets. The pipeline can therefore be reconstructed from fixed snapshots without repeating the initial Wikidata extraction. The release covers graph construction as well as inference.

4. Inference cost, accuracy, and cache reuse. The original experiment logs provide the following recorded means per problem; PASS@1 is macro-averaged across six benchmarks.

MethodTotal tokensLLM callsPASS@1
Direct1,202.171.0000.5753
CoT-ZeroShot1,439.811.0000.5933
SC@56,154.315.0000.6505
PAL1,048.461.0000.4907
Least-to-Most7,396.735.8260.5520
Code + Text + Sum.13,351.454.5370.6805
SBSC15,213.861.3420.3593
CoMAT2,275.371.0000.6093
TopoMath10,507.883.4730.7075

TopoMath uses fewer LLM calls but more tokens than SC@5. Relative to Code + Text + Sum., it achieves higher accuracy with fewer recorded tokens and calls.

We also evaluated SC@9 and re-evaluated Direct and TopoMath in a separate supplementary batch. SC@5 uses the first five candidates, fixed in advance, from the same nine-candidate pool. TopoMath achieves macro PASS@1 of 0.7663 versus 0.7458 for SC@9, using 17,451.35 versus 19,841.96 recorded tokens/problem: 2.05 percentage points higher accuracy with 12.05% fewer tokens. This comparison uses measurements from the supplementary batch only and characterizes the accuracy-token trade-off at similar, not identical, token budgets.

A cached scaffold can be reused for subsequent solves of the same problem, avoiding repeated concept extraction, graph retrieval, expansion, and assembly. Each solve still includes the scaffold in its prompt and may invoke verification and revision. If C_build is the initial construction cost and C_solve the per-solve cost, reuse across R solves yields an average of C_solve + C_build/R. With ten solves, only the construction component is amortized to one-tenth; different problems require their own scaffolds. Historical construction caches lack complete API-usage records, so an exact end-to-end cold-start cost or measured caching-savings percentage is not available.

Reviewer_KgwsOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TopoMath, a training-free inference-time pipeline for LLM mathematical reasoning. Offline, it builds a 2,346-node DAG of mathematical concepts from Wikidata’s subclass, part-of, and facet relations under six root domains. Online, an LLM extracts concept phrases from each problem and grounds them in the DAG, expanding it from Wikidata when needed. The method then renders a textual scaffold of anchors, root-to-anchor paths, shared ancestors, and neighbors, and adds it to the solver prompt. The solution is checked in two ways: SymPy executes the computational claims extracted from it, and a concept-membership check compares the concepts it uses against the scaffold. A failure in either check triggers one revision, which also sees an unguided solution as extra context. The evaluation covers six benchmarks (MATH-500, a 500-problem Omni-MATH subset, AIME 2025/2026, BRUMO 2025, HMMT 2026) and three backbones. The paper reports PASS@1 gains over direct inference (macro 0.575→0.708 with DeepSeekV4-Flash) and larger gains on DAG-Math-style process metrics (PRR 0.223→0.521). It also reports shorter reasoning graphs with fewer steps outside the final answer’s dependency ancestry.

Critical strengths
  • Simple, training-free, and modular design. The pipeline needs no fine-tuning. Its execution check uses deterministic SymPy rather than learned verifiers, so it can sit on top of any backbone available through an API. Graph construction, projection, and verification are cleanly separated, and each could be reused or replaced on its own.
  • Contamination-aware benchmarks and broad coverage. AIME 2026, HMMT 2026, and BRUMO 2025 make it less likely that gains come from memorized problems. The evaluation covers three backbones from two model families and seven inference-time baselines. PASS@1 improves over direct inference in all 18 backbone–benchmark pairs.
  • Evaluation beyond final-answer accuracy. The paper adopts the DAG-Math closure framework and adds Closure-AUC, which interpolates between PASS@1 and PRR. It also tests structural differences with paired Wilcoxon tests on problem-level means. The finding that answer-correct trajectories are much more often fully closed is interesting.
Critical weaknesses
  • The gains are not shown to come from the concept topology. On Omni-500, removing the scaffold lowers PASS@1 by only 1.6 points and PRR by only 0.6 points. That is about a fifth of the PASS@1 gain and almost none of the PRR gain over direct inference, and the variant with neither the scaffold nor verification is not reported. The verification only checks that a concept lies in the problem’s Wikidata neighborhood, not that it is applied correctly, and the revision step also sees an unguided solution, so its benefit may come from reconciling two samples. Without controls such as a scaffold the model generates itself or a random scaffold of the same size, the evidence supports the pipeline as a whole but not the claim that the topology drives the gains.
  • The comparisons are not controlled for runs, compute or generation budget. Direct and TopoMath are averaged over ten runs, but 55 of the 56 baseline entries on the competition sets are exact multiples of 1/30 (1/33 for HMMT 2026), which suggests single runs with a standard error of about ±9 points. Margins such as 0.708 vs. 0.680 macro PASS@1 over the strongest baseline are therefore within noise. TopoMath also makes at least four LLM calls per problem, yet the paper reports no token counts and no comparison against self-consistency or self-refinement at the same compute. Finally, thinking mode is disabled and outputs are capped at 4,096 tokens while baseline reasoning graphs are about 2.6× larger, so baseline outputs are more likely to be cut off before the answer, which could explain the very large Qwen3.5-27B gains (+33 to +39 points on AIME).
  • The process-level metrics favor short, rewritten solutions. Closure is the fraction of non-final steps that later steps reference, so short, linear write-ups score higher by construction. Reasonable exploratory steps in baseline solutions count as irrelevant, while TopoMath’s revised outputs are full rewrites that naturally drop dead ends. The reasoning graphs come from an LLM parser that has not been checked against human annotation, and parse failures score as closure 0, so long baseline outputs are penalized if they fail to parse more often. The PRR gains can only be credited to the topology after comparison with a length-controlled baseline, such as direct inference prompted for a concise, final-form solution.
Questions for authors
  • Missing related work. The method is close to LLM reasoning guided by knowledge graphs [1], and the paper should explain how it differs from that line of work. Prior work has also studied the two failure modes that motivate the paper: knowing a concept but failing to apply it [2, 3], and using a result whose premises do not hold [4, 5]. Discussing this work would clarify what checking concept membership can catch compared with checking preconditions directly.
  • Parser validity. What is the parse-failure rate for each method? Has the parser been checked against human annotations on a sample? How are closure and PRR defined for Self-Consistency@5, whose output is a majority vote over five trajectories?

[1] Sun et al. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. ICLR 2024.

[2] Wu et al. ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models. Findings of ACL 2024.

[3] Gao et al. CORE: Concept-Oriented Reinforcement for Bridging the Definition–Application Gap in Mathematical Reasoning. ICLR 2026.

[4] Ling et al. Deductive Verification of Chain-of-Thought Reasoning. NeurIPS 2023.

[5] Xu et al. Reliable Use of Lemmas via Eligibility Reasoning and Section-Aware Reinforcement Learning. ACL 2026.

4 comments
AuthorsOpenReview ↗

Overall Response

Thank you for the constructive feedback. Revisions are marked by blue font in the original manuscript for ease of presentation. Below, we summarize the supplementary experiments and methodological clarifications below; accompanying responses provide the detailed tables and protocols.

1. Contribution of concept topology and verification. In a new Omni-500 ablation batch, TopoMath achieves PASS@1 of 0.610, versus 0.588 with the same concepts and definitions presented without explicit relations, and 0.584 without scaffold guidance. With concept content and the review policy held fixed, the 2.2-percentage-point gap supports an incremental contribution of relational organization. We also add separate ablations of the two verifiers. Removing concept-membership verification yields PASS@1 of 0.582, a decrease of 2.8 percentage points; removing SymPy execution verification yields 0.596, a decrease of 1.4 points. Removing both verifiers also yields 0.596. The reference is an independently generated unguided solution from the same backbone, not a ground-truth solution. It aids revision without voting or direct reference-answer selection. Answer disagreement or execution failure triggers review of the primary solution; membership flags supply auxiliary feedback rather than independently triggering revision. These ablations examine scaffold representation and verification feedback within the shared review-and-revision framework.

2. Runs, inference cost, and output limits. We report recorded input/output tokens and LLM calls. In the original experiments, TopoMath averages 3.473 LLM calls and 10,508 tokens, versus 5 calls and 6,154 tokens for SC@5. In a separate batch re-evaluating Direct, Self-Consistency, and TopoMath, SC@5 uses the first five candidates from the SC@9 pool. TopoMath achieves macro PASS@1 of 0.7663 versus 0.7458 for SC@9, using 17,451 versus 19,842 tokens: 2.05 percentage points higher accuracy with 12.05% fewer recorded tokens. With Qwen3.5-27B's output cap increased to 16,384 tokens and other settings unchanged, TopoMath reaches 0.802 versus 0.683 for Direct, with gains on all six benchmarks.

3. Sensitivity to solution presentation. On Omni-500, we apply the same answer-preserving rewriting procedure to Direct's original outputs and TopoMath's outputs before final write-up refinement. The editor is instructed to preserve errors and unresolved gaps, receives no ground truth or verification feedback, and makes one attempt per eligible response. All conditions use the same parsing protocol and fixed reference DAG per problem. PRR increases from 0.204 to 0.346 for Direct and from 0.240 to 0.366 for TopoMath, while PASS@1 remains 0.570 and 0.610. The post-rewriting PRR gap is 0.020, with a problem-level paired-bootstrap 95% CI of [-0.020, 0.060]. However, this difference is not statistically significant, as the confidence interval includes zero.

4. Related work and positioning. We will expand the discussion of all five suggested papers. Think-on-Graph [1] explores KG entities and relations, whereas TopoMath constructs a problem-specific concept scaffold before generation and keeps it fixed during reasoning. ConceptMath [2] and CORE [3] provide context for concept knowledge and application; TopoMath supplies concept guidance at inference time without parameter updates. Deductive Verification [4] and lemma-eligibility reasoning [5] examine premises and applicability. Our membership check assesses scaffold inclusion, while SymPy checks selected executable claims. We position TopoMath as concept scaffolding with diagnostic revision, distinct from prerequisite or full-proof verification.

5. Parser validity and SC@5 evaluation. After format repair and retries, final parse-failure rates in the main DeepSeek experiments are Direct 0.05%, CoT-ZeroShot 0%, SC@5 0%, PAL 0.05%, Least-to-Most 0%, Code + Text + Sum. 0.13%, SBSC 0.13%, CoMAT 0%, and TopoMath 0%. Comparison with human annotations gives a mean absolute Closure error of 0.004 on the checked sample. For SC@5, we select the smallest sample_id among trajectories supporting the voted answer, without using correctness or closure scores. Closure is computed on this trajectory; PRR requires a correct voted answer and full closure. Unrecovered parses remain in the denominator with Closure and PRR contributions set to zero.

AuthorsOpenReview ↗

1. Contribution of Topological Structure and Role of Verification

Thank you for pointing out that the performance gains of the complete pipeline cannot be directly attributed to the concept topology. We agree that the contributions of additional concept information, explicit topological relations, and reference-assisted review and revision need to be distinguished. To address this concern, we conducted the following additional ablation experiments on Omni-500.

Because commercial model API services may change over time and outputs may vary across evaluation runs, we re-evaluated the complete TopoMath pipeline and all ablation variants, using the complete pipeline from this evaluation batch as the common reference. Consequently, the absolute scores below differ from those in the original manuscript. All ablation differences are calculated within this new batch, without mixing original and re-evaluated results. Delta denotes the absolute change in Pass@1 relative to the complete TopoMath pipeline in this batch.

MethodAblation settingPass@1Delta
TopoMathComplete pipeline0.610-
w/o Dual VerificationRemoves both verifiers0.596-0.014
w/o Topology ScaffoldRemoves scaffold guidance0.584-0.026
w/o Topological RelationsRemoves relations; retains concepts0.588-0.022
w/o Topology VerificationRemoves concept-membership checks0.582-0.028
w/o Execution VerificationRemoves SymPy verification0.596-0.014

Contribution of Topological Relations

The most direct control is w/o Topological Relations. This variant retains the same concepts and their available definitions, replacing only the structured scaffold with an unordered concept list. Pass@1 decreases from 0.610 to 0.588, a reduction of 2.2 percentage points. This result supports the incremental value of explicit relational structure over a concept list, distinguishing which concepts are supplied from how those concepts are organized. Removing scaffold guidance from the reasoning prompt altogether yields a Pass@1 of 0.584, a reduction of 2.6 percentage points relative to the complete pipeline, supporting the role of problem-relevant concept guidance in this setting.

Role of the Reference Solution in Revision

The reference solution is generated separately by the same backbone model, without access to the scaffold or the primary solution. When the final answers of the primary and reference solutions disagree, or execution verification reports a failure, the revision model receives the problem, the applicable guidance, the primary solution, verification feedback, and the full reference solution. It then performs one round of review and revision of the primary solution.

The scaffold-guided solution remains the subject of review throughout. The system neither performs answer voting nor programmatically selects the reference branch as the final output. The complete method and w/o Topological Relations retain the same reference-assisted revision strategy. Thus, the 2.2-percentage-point difference is observed under a shared revision mechanism. This comparison examines the contribution of explicit topological relations, rather than whether a reference solution participates in revision.

Role of Verification Feedback

Removing concept-membership checks reduces Pass@1 by 2.8 percentage points, whereas removing SymPy execution verification reduces it by 1.4 percentage points. Removing both verifiers, while retaining the scaffold and reference-assisted revision, yields a score of 0.596. The complete configuration achieves the highest score among these variants, supporting the use of verification feedback during review and revision of the primary solution.

The two verifiers serve different purposes:

  • Execution verification: SymPy checks extracted computational claims that can be formalized.
  • Topology verification: Checks whether concepts used in the reasoning belong to the problem-relevant concept set. This is a concept-membership check; it does not establish correct concept application, the satisfaction of theorem premises, or the validity of a complete deduction.

Concept-membership anomalies provide auxiliary feedback when execution failures or answer disagreements occur; they do not independently trigger revision. We will clarify this distinction in the Methods section of the revised manuscript.

Summary

These additional ablations examine the incremental value of explicit topological relations while retaining the same concept information and reference-assisted revision strategy. The results support combining structured concept guidance with verification feedback to improve the generation and revision of the primary solution.

AuthorsOpenReview ↗

2. Fairness of the Experimental Comparisons

Thank you for raising concerns about run counts, compute budgets, and output limits. We agree that shorter final solutions do not imply lower system costs. We added resource statistics, an SC@9 comparison, and Qwen3.5-27B re-evaluation with a 16,384-token output limit.

Because commercial model API services may change over time and outputs may vary across evaluation runs, we re-evaluated Direct, Self-Consistency, and TopoMath within a new batch. The resulting absolute scores differ from those in the original manuscript. All supplementary comparisons use results from this new batch, without mixing original and re-evaluated scores; resource statistics from the original experiments are reported separately. Re-evaluation has not yet covered every method–dataset combination. We will complete the remaining comparisons using consistent run counts and evaluation protocols and update the manuscript accordingly.

(1) Recorded resource usage

The following table reports the mean input tokens, output tokens, total tokens, and model calls per problem, together with macro-average Pass@1.

MethodInput tokensOutput tokensTotal tokensCallsPass@1
Direct120.151,082.021,202.1710.5753
CoT-ZeroShot156.781,283.031,439.8110.5933
SC@5600.775,553.546,154.3150.6505
PAL453.76594.711,048.4610.4907
Least-to-Most5,419.081,977.657,396.735.8260.552
Code+Text+Sum.9,978.723,372.7313,351.454.5370.6805
SBSC11,914.923,298.9415,213.861.3420.3593
CoMAT624.781,650.592,275.3710.6093
TopoMath6,613.323,894.5610,507.883.4730.7075

These results show that higher token usage does not necessarily correspond to higher accuracy. Nevertheless, we agree that such descriptive comparisons across methods cannot replace more direct budget-controlled experiments. For example, SC@5 makes more model calls but uses fewer total tokens than TopoMath. We therefore acknowledge the budget difference between the original SC@5 and TopoMath evaluations: this comparison alone cannot rule out benefits from additional computation. Conversely, Code+Text+Sum. and SBSC consume 13,351.45 and 15,213.86 tokens, respectively, both more than TopoMath, without achieving higher accuracy. Resource usage and performance therefore do not follow a simple monotonic relationship.

(2) Higher-budget Self-Consistency comparison

We added SC@9 and re-evaluated Direct and TopoMath in the same batch. SC@5 uses the first five candidates, fixed in advance, from the same nine-candidate pool. The per-dataset results are shown below.

Pass@1

DatasetDirectSC@5SC@9TopoMath
AIME 20250.7670.8000.8330.800
AIME 20260.7670.7670.8000.833
BRUMO 20250.8000.8330.8330.833
HMMT 20260.4850.3940.4550.576
MATH-5000.9320.9420.9500.946
Omni-5000.5700.6000.6040.610
Avg.0.72130.72270.74580.7663

Mean total tokens per problem

DatasetDirectSC@5SC@9TopoMath
AIME 20252,481.5312,733.0323,138.1020,657.40
AIME 20262,272.3012,169.3321,694.6318,608.30
BRUMO 20252,132.0311,227.4020,013.5716,590.80
HMMT 20262,900.0314,851.3927,234.7623,265.09
MATH-500767.843,697.166,649.576,418.46
Omni-5002,250.8111,306.2120,321.1519,168.07
Avg.2,134.0910,997.4219,841.9617,451.35

TopoMath achieves 2.05 percentage points higher macro-average Pass@1 than SC@9 while using 12.05% fewer recorded tokens. These results indicate that TopoMath's performance cannot simply be attributed to using more tokens than Self-Consistency.

(3) Output limits

An identical output cap can affect methods differently. We increased Qwen3.5-27B's limit to 16,384 tokens, keeping other settings unchanged and thinking mode disabled.

DatasetDirectTopoMathDelta
AIME 20250.6600.833+0.173
AIME 20260.7430.863+0.120
BRUMO 20250.7200.857+0.137
HMMT 20260.4330.624+0.191
MATH-5000.9380.950+0.012
Omni-5000.6020.682+0.080
Avg.0.6830.802+0.119

TopoMath retains higher Pass@1 on all six datasets, with an 11.9-percentage-point macro-average gain. These results show TopoMath's gains are not entirely attributable to a lower output limit constraining Direct.

AuthorsOpenReview ↗

Process-Metric Sensitivity, Related Work, and Parser Validity

3. Sensitivity of Process Metrics to Solution Presentation

Thank you for pointing out that the presentation of final solutions may affect process metrics. To examine this effect, we conducted a standardized rewriting control for Direct and TopoMath on Omni-500. We held the existing solutions and their final answers fixed as the source material, applied the same editing rules to both groups, and evaluated the reasoning graphs before and after rewriting using a consistent parsing pipeline.

Standardized Rewriting Rules and Experimental Setup

We used the same prompt to edit Direct's original responses and TopoMath's responses before its final write-up step. The rewriting model received only the problem, the existing solution, and the predicted answer. It was instructed to remove repetition and explicitly abandoned irrelevant exploration while preserving necessary derivations, existing errors, and unresolved gaps, without adding mathematical content or changing the final answer. Each eligible response was rewritten once. The original text was retained if it was incomplete or if the rewritten version failed the completeness or answer-consistency checks.

Both rewriting and parsing used Deepseek-V4.1-flash with temperature=0, thinking mode disabled, and an output limit of 16,384 tokens for each task. Both groups used the same parsing protocol and the same fixed reference DAG for each problem.

MethodStatusPass@1PRRClosure-AUCMean Closure
DirectBefore rewriting0.5700.2040.5130.851
DirectAfter rewriting0.5700.3460.5390.902
TopoMathBefore rewriting0.6100.2400.5600.889
TopoMathAfter rewriting0.6100.3660.5770.916

Standardized rewriting improved the process metrics for both groups without changing their final answers, indicating that these metrics are sensitive to solution presentation. After rewriting, TopoMath's PRR remained slightly higher than Direct's, but the difference was not statistically significant.

4. Related Work and Positioning

Thank you for highlighting these relevant studies. We agree that knowledge-graph-guided reasoning, mathematical concept application, and verification of reasoning premises all have substantial foundations in prior work. We will expand the discussion in the Introduction and Related Work sections and clarify TopoMath's connections to these studies, its differences, and the limits of its capabilities.

5. Parser Validity

Thank you for raising this concern. We examined the parsing logs from the main DeepSeek experiments across six datasets. After the prescribed format-repair and retry procedures, the final parse-failure rates were Direct (0.05%), CoT-ZeroShot (0.00%), SC@5 (0.00%), PAL (0.05%), Least-to-Most (0.00%), Code + Text + Sum. (0.13%), SBSC (0.13%), CoMAT (0.00%), and TopoMath (0.00%). We also compared parser outputs with human annotations on a sample of reasoning traces. The mean absolute error in Closure was 0.004, indicating close agreement in Closure estimation within the evaluated sample.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Area Chair and Reviewers,

Thank you again for your thoughtful feedback and suggestions.

We submitted our point-by-point responses and supplementary experimental results on Thursday, November 12. We have also updated the manuscript, with revisions and additions marked in blue for ease of review.

We would appreciate your feedback on whether these responses and additions address your main concerns, and whether any issues remain unresolved or require further clarification. We would be happy to discuss these points during the remaining discussion period.

Thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.