acceptodds
Under review as a conference paper at ICLR 2027

When Does Structured Knowledge Help Neural Theorem Proving?

Abstract

Does structured mathematical knowledge help LLMs prove theorems in Lean 4? If so, for which models, and does the answer vary by problem? Formal libraries such as Mathlib encode 285,000+ verified theorems with syntactic dependencies, but the semantic layer mathematicians rely on for discovery, such as analogies, generalizations, and cross-domain bridges, remains implicit. We introduce MathAgent, a system that builds this missing layer as a knowledge graph called MathKG, and uses it to augment LLM theorem provers. MathKG connects 364 Mathlib theorems and definitions by 9,434 typed semantic edges, inferred via LLM-based relation extraction anchored to verified Mathlib declarations. We run a controlled ablation across four augmentation modes (no external context, knowledge-graph context, Mathlib library retrieval, and both combined) and five models: the general-purpose Qwen3-8B/32B, their Lean-specialized derivatives Goedel-Prover-V2-8B/32B, and the proprietary Claude Sonnet 4.6. We evaluate on miniF2F, adding PutnamBench and MathOlympiadBench for the proprietary model. Three findings emerge. (i) Specialization dominates augmentation by an order of magnitude: Lean fine-tuning adds 33–38 percentage points of solve rate in every augmentation mode, and a specialized 8B model even beats a larger general one by 29–35 points in every mode; by contrast, no single augmentation mode improves solve rate by more than 3 points, so external knowledge does not substitute for competence in the weights. (ii) Augmentation's aggregate effect is capability-conditioned: knowledge-graph context helps small models but hurts large ones, with the specialized model gaining more relative to its general base at every scale. (iii) Yet across every model the augmentation modes solve different problems, so an oracle that selects the best augmentation mode per problem solves 6% to 58% more problems than the unaugmented prover (least for the proprietary model, most for a weak general one), revealing a complementarity effect that strengthens on harder problems. Together these results motivate adaptive strategies that select augmentation by model capability and problem. Code, data, and artifacts will be released upon publication.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
automated theorem proving, formal mathematics, mathematical reasoning, Lean 4, Mathlib, multi-agent systems, ablation study, knowledge graphs, retrieval-augmented generation, inference-time augmentation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_haVJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether structured mathematical knowledge can improve LLM-based theorem proving in Lean 4. It introduces MathKG, a knowledge graph of 364 Mathlib theorems and definitions connected by 9,434 typed semantic relations, and MathAgent, a multi-agent framework that uses the graph alongside Mathlib retrieval to support formal proof generation. Experiments on miniF2F, PutnamBench, and MathOlympiadBench show that Lean-specialized training substantially outperforms inference-time knowledge augmentation, while static augmentation provides limited and model-dependent benefits, sometimes even hurting stronger models. However, different augmentation strategies solve complementary problems: an oracle combining their successes achieves higher coverage, with particularly large relative gains on harder benchmarks. These findings suggest that the key challenge is not simply providing more knowledge, but adaptively selecting the most useful knowledge and augmentation strategy for each problem.

Critical strengths
  1. The paper introduces MathKG, a structured knowledge graph of mathematical concepts and theorems, integrated into a multi-agent framework for Lean 4 theorem proving.
  2. It systematically compares knowledge augmentation strategies across models with different capabilities, revealing that their effectiveness is model-dependent and can even be negative for stronger models.
  3. It demonstrates the complementarity of different augmentation strategies through oracle-union experiments, motivating future research on adaptive knowledge selection.
Critical weaknesses
  1. The benefits of knowledge augmentation are limited and inconsistent across models, with some stronger models experiencing performance degradation.
  2. The oracle-union results represent an upper bound rather than a practical adaptive selection method, leaving the key challenge of selecting the best augmentation strategy unresolved.
  3. The evaluation focuses primarily on established theorem-proving benchmarks, while the proposed open-ended mathematical exploration capabilities are not experimentally validated.
Questions for authors
  1. Why does knowledge augmentation help weaker models but hurt stronger models? English

The results show that knowledge augmentation can substantially improve weaker models but provides limited or even negative gains for stronger models. Can the authors analyze the underlying reasons for this difference? For example, is it due to knowledge redundancy, context interference, or differences in the models' ability to utilize structured knowledge?

  1. How does MathKG differ from existing graph-based theorem-proving methods? English

How does MathKG differ fundamentally from existing graph-based theorem-proving and premise-retrieval methods, such as LeanDojo/ReProver and recent graph-based retrieval approaches? Can the authors clarify the novelty of the proposed knowledge representation and provide direct empirical comparisons with these methods?

1 comment

We thank the reviewer for the insightful feedback and for their patience. Points shared with the other reviewers are answered in the general response.

The oracle. See point 1 of the general response: the union is the aggregation of pipelines that are all implemented and run, not a learned selector, and point 2 for the equal-budget comparison with independent restarts.

Relation to LeanDojo/ReProver and graph-based retrieval. ReProver retrieves premises for the current proof state with a dense retriever trained on premise usage, inside a tactic-level prover. MathKG is a different object used at a different point: named theorems and definitions with natural-language statements and verified Mathlib anchors, connected by LLM-inferred semantic relations under a typed ontology rather than by premise co-occurrence, and supplied once, before the first attempt, to an off-the-shelf prover. Recent graph-based retrievers (Khalov et al. and Ataeva et al.; KG-Prover) are closer in spirit and we cite them as such; our contribution relative to them is the cross-model comparison, not a retrieval method. On the empirical comparison: the Mathlib condition was deliberately lexical, and learned premise selection is out of the paper's scope (Section 5.3 and the revised Limitations section). A head-to-head with ReProver would change the prover and the proof procedure at once, not only the knowledge source, so it would not answer the paper's question; we leave it as future work.

Open-ended exploration. Agreed: the discovery loop (Explorer → Conjecturer → Prover → Integrator) is not evaluated in the paper, and the revised Limitations section says that all results are for targeted proving. Before submission we did run the system to open problems: open conjectures from the Formal Conjectures repository, 9 Erdős problems (pass@32) and 10 OEIS conjectures (pass@64) with Sonnet 4.6, and all 134 OEIS conjectures with Goedel-Prover-V2-32B (pass@64), under the four augmentation modes. What we learned there shaped our view of what open-ended proving needs: generating a whole proof per attempt is the wrong unit for these problems, because the retry loop gives no credit for the parts of an attempt that compile. That argues for stating the goal and its subgoals explicitly, keeping the parts that compile and retrying only the subgoals that fail. Supporting that direction means changing which part of the execution the system captures and iterates on, from whole proofs to subgoals, and since the targeted pipeline and its evaluation were already the heavy part of this work, we chose not to split the paper's focus.

Reviewer_hCPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when structured mathematical knowledge improves neural theorem proving. The authors introduce MathKG, a knowledge graph containing 364 Mathlib declarations and 9,434 semantic edges, and MathAgent, a multi-agent framework that augments Lean theorem proving with knowledge-graph context and Mathlib retrieval.

Experiments compare four augmentation modes across five models on miniF2F, with additional evaluations on PutnamBench and MathOlympiadBench. The results suggest that model specialization is more effective than knowledge augmentation, while different augmentation modes exhibit problem-level complementarity.

Critical strengths
  1. The paper provides a useful controlled comparison across model scales, specialization levels, and augmentation strategies. The matched Qwen3 and Goedel-Prover-V2 configurations help isolate the effects of Lean specialization, while the four augmentation modes provide insights into the effectiveness of external mathematical knowledge.

  2. The observation that knowledge augmentation can benefit smaller specialized models while providing limited or negative gains for stronger models is informative. The problem-level complementarity analysis also suggests opportunities for adaptive retrieval and augmentation strategies. The inclusion of significance tests and matched-attempt-budget oracle results strengthens the experimental presentation.

Critical weaknesses
  1. The paper attributes potential benefits to structured mathematical knowledge, but does not adequately isolate the contribution of graph structure from simply providing additional theorem descriptions. There are no comparisons against unstructured retrieval of the same nodes, randomized graph edges, or context-matched theorem lists. Furthermore, the 9,434 semantic edges are primarily LLM-inferred, without systematic independent validation of their mathematical correctness. Appendix D.3 reports that 11.4% of initially inferred edges violate basic type restrictions. Without stronger graph-quality evaluation and structural ablations, it remains unclear whether the knowledge graph itself provides meaningful advantages over ordinary contextual augmentation.

  2. The aggregate improvements from knowledge augmentation are small and inconsistent. For example, KG augmentation improves Goedel-8B from 42.0% to 44.9%, but reduces performance for Qwen3-32B and provides no statistically significant improvement for several other models. The paper emphasizes substantial gains from selecting the best augmentation mode per problem. However, this oracle assumes knowledge of which mode will successfully solve each problem, making it an optimistic upper bound rather than a deployable strategy. Although the matched-budget oracle analysis is helpful, the paper does not demonstrate a practical selector or compare against equivalent-budget independent baseline runs. Additionally, uncapped KG context causes overflow failures for Goedel-32B, complicating the interpretation of augmentation effects. These limitations weaken the central conclusion that structured knowledge provides substantial latent benefits for theorem proving.

Questions for authors
  1. Can the authors compare MathKG against unstructured retrieval of the same nodes, randomized edges, or similarly sized context without graph relations? How much of the improvement specifically comes from graph structure?

  2. Has the semantic accuracy of the LLM-generated edges been evaluated by human experts? What are the precision rates for different relation types, especially cross-domain bridges and proof dependencies?

  3. Can the authors implement an adaptive selector that predicts the appropriate augmentation mode before proof generation? How does it compare with the oracle and equivalent-budget independent baseline runs?

  4. How do results change under matched context lengths, token budgets, and Lean/Mathlib versions? Can the authors quantify the effects of context overflow and stochastic variation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer hCPb (1/2): Q1 graph structure versus context, Q2 edge accuracy

We thank the reviewer for a summary that states the paper's claims as we intended them, for the four questions, which we answer in order, and for their patience. Points shared with the other reviewers are answered in the general response.

Q1, graph structure versus additional theorem descriptions. We agree that the submission does not isolate the contribution of graph structure, and the measurements made for this response change what there is to isolate. The KG block is built from seeds chosen by the evaluated model, which the submission did not log. Re-running the Explorer on a 61-problem sample stratified by domain label, Qwen3-32B returns seeds on 56 problems, Goedel-32B on 26 and Goedel-8B on none. Goedel-8B's with_kg prompts therefore carried no graph content, and its +14, the paper's one significant gain with the graph, is not an effect of the graph. We withdraw it: it is within the spread of four independent prover_only chains (190 to 202 problems at 8 attempts, against 194 and 206 for the first 8 attempts of the paper's prover_only and with_kg runs). So no improvement is left to attribute to graph structure.

The one KG effect the data support is the loss for Qwen3-32B (62 → 48, p=.007), the model that receives the block on almost every problem. It occurs at the first attempt, the only one with the block in the prompt: first-attempt solves fall from 42 to 28, where they are 44 with the much shorter Mathlib block and 36 to 44 in four independent prover_only chains. Whether this loss comes from the block's length (a mean of about 19,400 tokens), from its content or from its typed relations is exactly what the reviewer's controls would separate: the same nodes without relations, the same number of random nodes, and a capped block. We designed these three controls but could not run them within the discussion period, so the question of what graph structure contributes remains open in this paper, and the revision says so.

Q2, semantic accuracy of the edges. We audited a stratified sample of 100 inferred edges, reading each edge with both node statements and the extractor's reasoning, and we release the per-edge verdicts with the revision. We scored relevance: whether the two nodes share a genuine conceptual link, whatever the edge type, with generic links such as "both live in a ring" counted as not relevant. Under that definition 74% of the sampled edges connect relevant pairs; under a more permissive one that also accepts links mediated through a third concept, 84%. Relevance is the criterion retrieval depends on: the Explorer follows every edge of a seed regardless of its type, using the type only to extend candidates across domains and as a label on the context line. A mathematician is now going through the same sample for the accuracy of the relationship types, and we will update the numbers here and in Appendix D.4 when that pass is complete. One clarification on the 11.4%: those 1,041 edges were removed in post-processing (Appendix D.3), so the released graph contains no type violations; the figure measures the raw extractor, not the graph that the Prover saw.

AuthorsOpenReview ↗

Response to Reviewer hCPb (2/2): Q3 practical selector, Q4 token budgets, toolchains, overflow, variation

Q3, a practical selector versus the oracle. See point 1 of the general response (the union as the aggregation of pipelines that are already implemented, and why we did not find a per-problem selector) and point 2 for the equal-budget independent baseline. In short: at the same budget of four chains of 8 attempts, four independent prover_only chains solve 79, 229, 258 and 360 problems for Qwen3-32B, Goedel-8B, Goedel-32B and Sonnet, against 79, 232, 259 and 370 for the union of the four pipelines. For the open-weight models the gain of the union is reproduced by restarts.

Q4, matched context lengths, token budgets, toolchain versions, overflow, and stochastic variation.

Context lengths. We did not log the blocks in the original runs, so we re-ran the Explorer alone on a 61-problem sample (table in our reply to Reviewer R2By, Q3). When a KG block is present it holds a median of about 115 statements and about 20,000 tokens, against a Mathlib block capped at 8/5/2 declarations per search term. It is not always present: Qwen3-32B receives it on 56 of the 61 problems, Goedel-32B on 26 and Goedel-8B on none. We could not run a capped-block control within the discussion period.

Token budgets. Each cap was set above the longest proof that model produced in our setting. As an equal-cap check, on a fixed set of 30 problems Goedel-32B solves 12 at its 4,096 cap and 12 at Goedel-8B's cap of 16,384 (reply to Reviewer R2By, Q1).

Lean/Mathlib versions. Within each model all four modes ran on one toolchain, so every within-model delta (Table 3) and every union (Tables 4 and 12) is version-matched. Across models there are two toolchains, both stated in Appendix F: the May 2026 runs (Sonnet 4.6, Qwen3-8B) used Lean 4 and Mathlib v4.30.0-rc2, the August 2026 runs (Goedel-8B, Goedel-32B, Qwen3-32B) v4.32.0-rc1. The only cross-model statement spanning both is the 8B specialization gap, and 35 points is far outside any plausible toolchain effect. The revision states this in Section 5.1.

Context overflow. This is quantified in the paper (footnote 6 and Appendix I) and we restate it here. Overflow occurred only for Goedel-32B, whose server exposed a 40,960-token window, and only in the two KG modes: 10 problems under with_kg and 28 under full_system, counted as failures. Excluding them, the rates are 49.0% and 45.2% instead of 48.0% and 42.6%; the with_kg delta against prover_only moves from −3 to +1, and the full_system delta from −29 to −6 against prover_only and from −27 to −3 against with_mathlib. Of the overflowed problems, prover_only had solved 4 of the 10 and 23 of the 28, so overflow is the main reason the Goedel-32B full_system row looks worse than its with_mathlib row.

Stochastic variation. The four independent prover_only chains of point 2 in the general response give the chain-to-chain spread at 8 attempts: 56 to 65 problems for Qwen3-32B, 190 to 202 for Goedel-8B, 223 to 231 for Goedel-32B and 318 to 330 for Sonnet. The first two Sonnet chains disagree on 43 problems (22 and 21 each way) while agreeing on the count. Differences between modes of this size should be read against that spread.

Reviewer_R2ByOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether supplying structured mathematical knowledge at inference time helps LLM provers in Lean 4, and for which models. The authors build MathKG, a graph of 364 Mathlib-verified theorems and definitions in seven undergraduate domains, with 9,434 typed edges (8,026 inferred by Claude Sonnet 4.6 over all ordered node pairs, plus 1,408 auto-generated inverses) in eight relation types. They also describe MathAgent, a four-agent system (Explorer, Conjecturer, Prover, Integrator). Only the Explorer and the Prover are used in the experiments. The evaluation is a controlled ablation of four context modes (none, KG context, lexical Mathlib name retrieval, both) across a 2×2 of Qwen3-8B/32B versus Goedel-Prover-V2-8B/32B, plus Claude Sonnet 4.6. The paper reports three findings:

  1. Lean specialization adds 35.2 points at 8B and 35.9 points at 32B, while no augmentation mode moves any model by more than 3 points.
  2. KG context helps Goedel-8B (+14 problems, McNemar p=.029) but hurts Qwen3-32B (−14, p=.007), so its effect depends on the model.
  3. The union of problems solved by the four modes exceeds prover_only by 18 to 36 problems. The paper calls this complementarity and presents it as motivation for adaptive augmentation.
Critical strengths
  1. The experimental design is careful about the comparisons it makes. Each general model is paired with its Lean-specialized derivative at the same scale, prompts are shared across modes, and every Table 3 delta comes with an exact McNemar test. The appendix also addresses the confounds a reader would raise first. This makes the aggregate deltas trustworthy as measurements of this specific pipeline.

  2. The question is useful for the community and the answer is informative even though it is mostly negative. For the configurations tested, knowledge in the weights dominates knowledge in the prompt, and KG context can significantly hurt a stronger model. This is a useful caution for the many recent agentic provers that add retrieval or memory layers without isolating their effect.

Critical weaknesses
  1. The provers are run far from their standard settings, which undermines the cross-model conclusions. Thinking mode is disabled and outputs are capped at 4,096 tokens for Goedel-32B, 16,384 for Goedel-8B and 2,000 for the other models (Appendix F). Goedel-32B then solves only 48.6% of miniF2F at pass@32, far below what the Goedel-Prover-V2 authors report, and the paper does not address this gap. Finding (ii) is most exposed, since it compares a 4K-capped 32B model with a 16K-capped 8B model.

  2. The findings are attributed to "structured knowledge" without the controls that separate content from length or restart effects. The KG context is uncapped and overflows the context window on 10 and 28 Goedel-32B problems, yet there is no length-matched random-context control. The oracle unions four independent 8-attempt chains, but the matching baseline of four restarted prover_only runs is not reported; under the matched budget the Sonnet gain shrinks from +23 to +5 problems. The graph itself is LLM-built with no human check of edge precision.

Questions for authors

The following clarifications could change my assessment.

  1. Why was thinking mode disabled for Goedel-Prover-V2 and Qwen3, and how do your prover_only numbers compare with the officially reported miniF2F pass@32 of Goedel-Prover-V2 under its recommended settings? Do findings (i) and (ii) hold when Goedel-8B and Goedel-32B are run with the same, long generation budget?

  2. What is the matched-budget union of four independent prover_only runs (8 attempts each, different seeds) for each model? How does it compare with the 51, 79, 232, 259 and 370 problems in Table 12?

  3. What is the average and maximum token length of the KG context block per model, and what happens with a length-matched random-context control or a capped KG context?

  4. Has any sample of the 8,026 LLM-inferred edges been checked by a mathematician for correctness? What is the estimated edge precision, in particular for cross-domain-bridge?

  5. For the 175 miniF2F problems that fall back to the whole graph, does the KG effect differ from that on problems with a matched domain?

In addition, as a suggestion that does not affect my score, the paper would benefit from a limitations section. It currently has none, although several of the issues above are acknowledged only in footnotes and appendices.

2 comments
AuthorsOpenReview ↗

Response to Reviewer R2By (1/2): Q1 thinking mode, official numbers and caps; Q2 matched-budget control

We thank the reviewer for the careful reading and their patience. The points shared with the other reviewers (the union of pipelines, the independent prover_only chains, and the limitations section) are in the general response. Two of the controls the reviewer asked for change the paper's conclusions, and we report them as they came out.

Q1a, why thinking was disabled. When we set up the open-weight runs we compared the two settings on a small pilot set, and the setting without thinking performed better in our pipeline, so we used it for all four open-weight models. For this response we repeated the comparison for the two Goedel models on a fixed set of 30 miniF2F problems (6 per label group): prover_only, the same prompts and 8-attempt budget, and the cap at 16,384 in the thinking setting so that the reasoning trace fits. Problems solved, of 30:

thinking off, 4,096thinking off, 16,384thinking on, 16,384
Goedel-8Bnot run (its cap is 16,384)910
Goedel-32B121211

In our pipeline neither thinking nor the cap changes the count by more than one problem on this set.

Q1b, comparison with the official Goedel-Prover-V2 numbers. Goedel-Prover-V2 reports 84.6% (8B) and 88.1% (32B) pass@32 on the 244-problem miniF2F test split, with 32 independent samples per problem, replies of up to 32K tokens that open with a natural-language proof plan, and no compiler feedback (90.4% for the 32B model with self-correction). Our protocol differs in four ways: thinking off and a prompt that asks for Lean code only, which removes the proof-plan stage the models were trained with; one chain of 32 error-guided refinements at temperature 0.3/0.2 instead of 32 independent samples; all 488 miniF2F problems rather than the 244 test problems; and the 4,096-token cap for the 32B model. On the test split alone our prover_only gives 96/244 (39.3%) for Goedel-8B and 106/244 (43.4%) for Goedel-32B, so the problem set is not the explanation, and on the 30-problem set above neither the cap nor thinking within our prompt is. The sampling scheme does matter: four independent chains of 8 attempts solve 229 (8B) and 258 (32B) of the 488 problems, where one chain of 32 attempts solves 205 and 237 (general response, point 2). The paper made no comparison with the official numbers, but the reviewer is right that the gap must be stated and explained, and the revised Section 5.1 does so.

Q1c, equal generation budgets and findings (i) and (ii). On the equal-cap question, Goedel-32B solves 12 of the 30 problems at its 4,096 cap and 12 at Goedel-8B's 16,384 cap, so the cap asymmetry between the two Goedel models does not affect prover_only on this set. Finding (ii) does not stand, for a different reason, described under Q3: Goedel-8B's with_kg prompts carried no KG block, so its +14 is not an effect of the graph. We withdraw (ii) as stated; the KG effect that remains is the loss for Qwen3-32B.

Q2, matched-budget union of independent prover_only runs. Done for four of the five models, exactly as the reviewer describes; the table is in point 2 of the general response. For Qwen3-32B, Goedel-8B and Goedel-32B the union of four independent prover_only chains matches the union of the four pipelines (79 against 79, 229 against 232, 258 against 259); for Sonnet it is 10 lower (360 against 370). So for the open-weight models the gain of the union is reproduced by restarts.

AuthorsOpenReview ↗

Response to Reviewer R2By (2/2): Q3 context length, Q4 edge precision, Q5 whole-graph problems, limitations

Q3, length of the KG context block and length controls. We did not log the blocks in the original runs. For this response we re-ran the Explorer alone on a 61-problem sample stratified by domain label, for the three models we could serve again (tokens counted with each model's tokenizer):

blocks returned, of 61statements per block, mediantokens: mean / max
Qwen3-32B5611919,400 / 33,100
Goedel-32B2611420,600 / 40,600
Goedel-8B0nonenone

So the block is long and uncapped when it is present. The seeds are selected by the evaluated model, and the Lean-specialized models frequently return no usable seed list: Goedel-32B on 35 of the 61 problems and Goedel-8B on all of them (in the replies we inspected, it answers the selection prompt with a proof of the problem). Goedel-8B's with_kg prompts therefore carried no graph content, and its +14 is not an effect of the graph; we withdraw that reading. It is within the spread of independent runs: four independent prover_only chains of 8 attempts solve 190 to 202 problems, against 194 and 206 for the first 8 attempts of the paper's prover_only and with_kg runs. The model on which the reviewer's two controls are informative is Qwen3-32B, which receives the block and loses 14 problems with it, at the first attempt (42 → 28 first-attempt solves; our reply to Reviewer haVJ, Q1). We could not run the length-matched random-context and capped-context controls within the discussion period, so we cannot say whether that loss comes from the block's length or its content.

Q4, edge precision. We audited a stratified sample of 100 inferred edges, reading each edge with both node statements and the extractor's reasoning, and we release the per-edge verdicts with the revision. We scored relevance: whether the two nodes share a genuine conceptual link, whatever the edge type, with generic links such as "both live in a ring" counted as not relevant. Under that definition 74% of the sampled edges connect relevant pairs; under a more permissive one that also accepts links mediated through a third concept, 84%. Relevance is the criterion retrieval depends on: the Explorer follows every edge of a seed regardless of its type, using the type only to extend candidates across domains and as a label on the context line. A mathematician is now going through the same sample for the accuracy of the relationship types, and we will update the numbers here and in Appendix D.4 when that pass is complete.

Q5, the 175 problems that fall back to the whole graph. A clarification first: for these problems (identifiers naming AMC/AIME, IMO or a technique rather than a topic) the graph is not absent; the Explorer's seed candidates are all 364 nodes instead of a domain subset, so the seeds are chosen with less prior relevance. Splitting the KG effect by group (solves under prover_only → with_kg; two-sided exact McNemar p on the discordant problems):

175 whole-graph problems313 topic-matched problems
Qwen3-8B3 → 3 (0)30 → 32 (+2, p=.79)
Qwen3-32B6 → 3 (−3, p=.25)56 → 45 (−11, p=.03)
Goedel-8B43 → 42 (−1, p=1.0)162 → 177 (+15, p=.004)
Goedel-32B53 → 50 (−3, p=.63)184 → 184 (0)
Sonnet 4.698 → 88 (−10, p=.01)267 → 270 (+3, p=.58)

The effect does differ, in where the losses are. Sonnet's loss is confined to the whole-graph problems, and Qwen3-32B loses in both groups, significantly on the topic-matched ones. The Goedel-8B row is not a KG effect: its prompts carried no KG block (Q3), so its +15 is variation between runs. For Goedel-32B the block was returned on 22 of the 39 sampled topic-matched problems and on 4 of the 22 whole-graph ones. This table goes into Appendix C with a sentence in Section 5.3.

Limitations section. Added in the revised PDF, as described in the general response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.