acceptodds
Under review as a conference paper at ICLR 2027

PatternMiner: Automated Mathematical Discovery Through LLM-Guided Training and Interpretation of Small Models

10 views

Abstract

A large body of work in AI for math focuses on two approaches to mathematical discovery: the use of large language models (LLMs) to resolve well-specified conjectures and the use of small, narrow models for more open-ended, human-in-the-loop exploration. LLMs draw on broad mathematical knowledge to produce human-readable arguments, while small models support example-driven discovery by detecting patterns in computed examples. We present PatternMiner, a system in which an LLM agent generates exact datasets for a chosen problem, trains small models, and interprets what the models learn. The agent formulates conjectures, searches for counterexamples that prompt revisions, and uses hard-negative mining to select training examples that satisfy the current necessary conditions but fail the target property. The agent attempts proofs of conjectures that survive these tests. Applied to five problems in algebraic combinatorics, PatternMiner proves Schubert positivity and projection theorems, a character-row evaluation formula, restrictions on Kazhdan–Lusztig combinatorial invariance, and Schur positivity for a family of immanant coefficients; it also recovers classical Kronecker bounds.

Primary area:
interpretability and explainable AI
Keywords:
AI for math, mathematical discovery, interpretability, combinatorics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CYGBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is concerned with the problem of discovering novel conjectures and proving them. The authors propose a system called PatternMiner, in which they use an LLM agent on the following workflow: study a problem, propose conjectures, generate datasets covering the conjecture, train models on these datasets, interpret the model to find new conjectures, test the conjecture, and attempt proofs of promising candidate conjectures. The authors apply their system to algebraic combinatorics where they find their system rediscovers some known conjectures, as well as proposes and proves new ones.

Critical strengths

While most work takes aim at proving existing conjectures and improving existing results, I think tackling the problem of finding new conjectures is a relatively new and exciting direction. I commend the authors for choosing this problem. While I’m not overly familiar with the related literature here, PatternMiner does look like a novel application of LLM agents to synthesise / accelerate parts of existing strategies for conjecture search. The idea of using agents to automate the process as a pipeline is interesting and valuable.

Critical weaknesses

W1: An important criteria for a new conjecture to be valuable is for it to be interesting to the community. It is certainly possible to discover new conjectures trivially that have no meaningful impact or are not interesting. The authors have not discussed whether their newly discovered conjectures satisfy this, and have not attempted to show, e.g. through qualitative studies or some other method, that these conjectures are valuable. This is a significant concern in evaluating the results of PatternMiner. Unfortunately, as I am not a mathematician, I am unable to verify the results or determine if they’re non-trivial and interesting.

W2: Along the same line, rediscovering existing conjectures does not necessarily show that the PatternMiner system has worked. Unfortunately, these existing results may have been seen by the agent in pre-training or mid-training, confounding the interpretation of rediscovered conjectures. Is there anything the authors can say / demonstrate to show that PatternMiner is responsible for these rediscoveries rather than it being driven by the LLM’s prior knowledge of the existing conjecture?

W3: PatternMiner is very poorly described and definitely is not reproducible at the moment. Section 3 provides a short and vague description of how the system works, while the Appendices omit any concrete details. I would like to know more about the prompts used, the tools used, and the limits of the kinds of conjectures the system can handle. Furthermore, this is a serious weakness as no code has been submitted. Therefore, from the paper, the appendices, and the lack of code, there is no detailed explanation of the method.

Questions for authors

Q1: Do the authors have any thoughts about applying recent work on using LLM agents for mechanistic interpretability in PatternMiner? For example, [A] claims to have found that most neurons in even very large neural networks have a single job, so very deep networks may be more interpretable than previously thought.

[A] Cammarata et al. 2026. https://already-clean.goodfire.com/

2 comments

Thank you for the thoughtful review and for recognizing conjecture generation as an important research direction. Your comments helped us clarify the scope of our claims and improve both the presentation and documentation of PatternMiner.

Mathematical value of the results. We agree that a conjecture is not valuable simply because it is new. Our motivation is that, as AI systems become better at proving well-specified statements, finding meaningful statements to investigate also becomes increasingly important. PatternMiner is therefore not designed to generate as many new conjectures as possible. Instead, it packages an end-to-end pipeline documenting how conjectures are proposed, tested, refuted, refined, and sometimes proved.

In the revisions, we specifically distinguish between new conjectures, rediscovered results, refuted conjectures, and general statements that were ultimately proved. The strongest outcomes included not only observations and proposed conjectures, but also general mathematical statements that PatternMiner formulated and proved. In one investigation, PatternMiner proved that a natural family of existing tests will always miss certain cases, regardless of how large the mathematical objects become. In another, it proved two previously proposed conjectures. The remaining investigations produced a more efficient way to calculate a mathematical quantity and extended a known result to a broader family of objects. Complete mathematical statements and proofs are provided in Appendix B.

Regarding the research topic, we selected problems from Stanley’s survey because they are well-recognized research problems with several meaningful directions of investigation. However, we agree that this does not replace an independent evaluation of how interesting the results are to the broader mathematical community.

Rediscovery and possible pretraining exposure. We agree that the LLM may have encountered the known results during training, and we cannot determine retrospectively how much its prior knowledge contributed to their rediscovery. For this reason, we do not use rediscovery as evidence that PatternMiner independently discovered those results.

To provide a more informative comparison, we added direct LLM baselines in Section 5 and Appendix C. The direct agents received the same five broad problems and tools for exact computation, but no trained predictor or PatternMiner workflow. They still made useful progress, although their conclusions generally covered more restricted cases or produced less efficient constructions. This helps show what the agents could accomplish without the small-model workflow, although it does not rule out the influence of pretraining.

System description and reproducibility. We agree that the original description was too vague. At a high level, PatternMiner is a markdown-based workflow for a coding agent rather than a single trained model or fixed orchestration program. The workflow contains five prompts, four record templates, and three method guides. The prompts guide the agent through the overall research process, experiment design, model training and interpretation, and conjecture development. The record templates preserve the project’s current state across sessions and document human interventions. The method guides provide additional instructions for model selection, interpretation, and reviewing relevant literature. Section 3 now summarizes this structure and explains each step in more detail. In addition, the anonymous repository provides the exact prompts, templates, guides, experiment records, and code. The models and budgets used for the new raw LLM baselines and controlled interpretation ablation are documented separately.

LLM agents for mechanistic interpretability. PatternMiner already uses an LLM agent to investigate how a trained model arrives at its predictions. In one of our studies, the agent tested how the model’s answers changed when specific internal components were modified. It then used those changes as evidence for constructing a mathematical rule.

We agree that applying similar methods to larger models is a promising direction. We focused on small models because it is easier to inspect their internal behavior, test whether an explanation is accurate, and connect what the model has learned to a precise mathematical statement. It is also immensely less costly than larger models, which is of high value to the open-source academic community.

Thank you for your response. Unfortunately, given how close we are to the discussion deadline, I have not had sufficient time to carefully evaluate the new material. I will take the response into account when finalising my assessment, but I can not provide further feedback within the discussion period.

Reviewer_ZT2dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents PatternMiner, where an LLM is used to train smaller models on datasets constructed from a problem. The LLM then examines and interprets the learned characteristics of these smaller models. This approach is applied to five algebraic combinatorics problems, where it makes progress on four. The paper does a close examination of interpretability and interventions.

Critical strengths

S1. The paper presents a strong general idea: for a wide variety of problems, one can construct a dataset of examples, train a predictor on the examples, and interpret the predictor. PatternMiner automates this process with the help of an LLM.

S2. While I cannot evaluate the mathematics myself, it seems that the results are substantial for the problems where the method makes some progress. This is a good general framework for making progress on a wide variety of problems.

Critical weaknesses

W1. The paper needs to have an ablation on what happens if no predictor is trained, and the LLM is given similar resources. This is not breaking, but it would be useful to have a fair comparison.

W2. There needs to be more comprehensive documentation of the LLM research system used (prompts, orchestration frameworks etc.)

W3. I believe the writing quality across the paper needs to be a lot better. The quality of exposition currently is not satisfactory, and it would be good to do a clean pass to make it better.

Questions for authors

Q1. What exactly constitutes the PatternMiner system? Please include prompts, orchestration framework, LLM versions, context handling etc..

1 comment

Thank you for the encouraging assessment and constructive suggestions. We revised the paper to address all three concerns.

Comparison without predictor training. We agreed that the original paper was missing a comparison showing what an LLM agent could do without training a predictor. We therefore added direct LLM baselines in Section 5, Appendix C, and Table 22. Two raw LLM agents received the same five broad problems and tools for exact computation, but no trained predictor or PatternMiner workflow. The agents still made useful progress, but their results generally applied to more restricted mathematical cases or gave less efficient constructions than those from the full PatternMiner investigations. We will not overstate this comparison: since every single run is expensive and not exactly replicable, we did not fully match the budgets, context, tools, and human guidance.

What constitutes PatternMiner? At a high level, PatternMiner is a markdown-based workflow for a coding agent rather than a single trained model or fixed orchestration program. The workflow contains five prompts, four record templates, and three method guides. The prompts guide the agent through the overall research process, experiment design, model training and interpretation, and conjecture development. The record templates preserve the project’s current state across sessions and record human steering. The method guides provide additional instructions for model selection, interpretation, and reviewing relevant literature. Section 3 now summarizes this structure and explains each step in more detail. In addition, the anonymous repository contains the exact prompts, templates, guides, experiment records, and code.

Writing and exposition. We went through the paper and made several changes to improve its presentation. We shortened the Method section, added a general-audience introduction to algebraic combinatorics, explained why the five problems were selected, and reorganized the analysis around a few recurring discovery patterns. We also added clearer summaries of the mathematical outcomes and expanded the experimental and system documentation in the appendices.

We appreciate these suggestions. In particular, the raw LLM comparison and expanded system description allowed us to explain more clearly what PatternMiner adds beyond an LLM agent equipped with mathematical software.

Reviewer_tsodOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents PatternMiner, a system for mathematical conjecture generation. The system iterates through four stages: (1) translating a mathematical question into a prediction task with exactly computable labels; (2) training small models on the data generated; (3) interpreting the learned computation; and (4) testing candidate conjectures by exact computation, mining hard negatives, and feeding revised examples back into training. The authors then evaluate this pipeline on five problems in algebraic combinatorics. The reported outcomes include rediscoveries of known results and new conjectures.

Critical strengths
  • The closed-loop design of PatternMiner is generally reasonable. It could discover both known results and new conjectures.

  • The five problems studied in the experiments are explained in detail. More details are provided in the appendix.

Critical weaknesses
  • The real-world impact of the proposed pipeline is unclear. It can indeed discover some conjectures or theorems. But how would that benefit the real world or math research is unclear, especially considering that the current experiments are only evaluated on five algebraic combinatorics problems.

  • The position of this work against existing literature is not clear. How would the proposed PatternMiner perform against related methods?

  • Some key designs are not well-evaluated. The paper argues that PatternMiner automates a loop previously run with human choice of data, models, and interpretation. But there is no controlled ablation showing that the interpretation component or the hard-negative-mining component is necessary.

  • How did the authors check the proofs is not clear. For example, who verified the proofs, how rigorously (e.g., is it hand-checked or verified in an interactive theorem prover)?

  • The paper is difficult to follow. Five problem studied are all from (a subfield of) algebraic combinatorics. Interpreting the usefulness of the generated conjectures is not easy for general audience.

Questions for authors
  1. How did the authors check the proofs?
  2. What are the real-world benefits?
1 comment

Thank you for the careful review. The revised paper addresses several of your concerns directly.

Mathematical impact PatternMiner is intended to support mathematical research. As AI systems become better at proving well-specified statements, finding good statements to prove becomes harder and harder, so for this project we are aiming to provide a solution for proof abundance and conjecture scarcity. Across five research-level problems, PatternMiner produced checkable constructions, algorithms, conjectures, and proofs. Table 1 and Section 4 summarize the results, with complete statements and proofs in Appendix B. We agree that five problems from one subfield do not demonstrate general applicability, and we now state this limitation explicitly in our paper. We hope that the present work will inspire research in this direction that is orthogonal to purely LLM-based methods for mathematical discovery.

Position relative to existing methods. We agreed that the original paper was missing an LLM-only comparison, so we added one in Section 5, Appendix C, and Table 22. Two raw LLM agents received the same five broad problems and tools for exact computation, but no trained predictor or PatternMiner workflow. They still made useful progress, but their results generally applied to more trivial mathematical cases or provided less efficient constructions than those from the full investigations. We do not want to overstate this comparison: since every single run is expensive and not exactly replicable, we did not fully match the budgets, context, tools, and human guidance.

We also clarified that FunSearch and AlphaEvolve search over LLM-generated programs, whereas PatternMiner uses patterns learned by small models to choose and refine mathematical directions. We haven't run a head-to-head comparison with these systems, again due to budget constraints, and we stated this limitation more explicitly. We reiterate our hope that the present work will inspire research in this direction that is orthogonal to purely LLM-based methods for mathematical discovery.

Component ablations. To better understand the upside of mechanistic interpretation, we gave two agents the same examples, tools, model outputs, and sets of attention heads from the small transformer. The only difference was that one agent could also see how changing individual attention heads affected the model’s predictions. Both agents were able to develop useful mathematical rules, but the intervention-guided agent performed better on unseen examples, succeeding on 214 of 249 held-out pairs compared with 174 of 249 for the outputs-only agent. Therefore, our takeaway is not that interpretation was necessary for finding a useful rule. Instead, it gave the agent additional evidence that helped it identify more relevant model components and construct a rule that generalized better.

We also looked at what happened when hard-negative mining was removed from training. The agent still found six difficult vanishing examples, but the investigation stopped with some individual counterexamples and an open conjecture. In comparison, the full investigation eventually reached a recursive construction proving that no fixed number of retained dimensions is sufficient. This is not a perfectly controlled comparison because the two investigations later followed different paths and received different human guidance. Still, the results suggest that mining difficult examples helped move the investigation beyond isolated cases and toward a more general mathematical result.

Proof verification. Two senior PhD students in algebraic combinatorics manually checked the proofs. Each proof also underwent adversarial review by a separate agent, which we treat as an additional check. The proofs have not been formalized in an interactive theorem prover, again due to budget constraints.

Accessibility. We added an explanation of why these problems were selected, a general-audience introduction to algebraic combinatorics, and clearer summaries of the main mathematical outcomes. We also reorganized Section 5 around recurring discovery patterns to make the conclusions easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.