- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper is concerned with the problem of discovering novel conjectures and proving them. The authors propose a system called PatternMiner, in which they use an LLM agent on the following workflow: study a problem, propose conjectures, generate datasets covering the conjecture, train models on these datasets, interpret the model to find new conjectures, test the conjecture, and attempt proofs of promising candidate conjectures. The authors apply their system to algebraic combinatorics where they find their system rediscovers some known conjectures, as well as proposes and proves new ones.
Critical strengths
While most work takes aim at proving existing conjectures and improving existing results, I think tackling the problem of finding new conjectures is a relatively new and exciting direction. I commend the authors for choosing this problem. While I’m not overly familiar with the related literature here, PatternMiner does look like a novel application of LLM agents to synthesise / accelerate parts of existing strategies for conjecture search. The idea of using agents to automate the process as a pipeline is interesting and valuable.
Critical weaknesses
W1: An important criteria for a new conjecture to be valuable is for it to be interesting to the community. It is certainly possible to discover new conjectures trivially that have no meaningful impact or are not interesting. The authors have not discussed whether their newly discovered conjectures satisfy this, and have not attempted to show, e.g. through qualitative studies or some other method, that these conjectures are valuable. This is a significant concern in evaluating the results of PatternMiner. Unfortunately, as I am not a mathematician, I am unable to verify the results or determine if they’re non-trivial and interesting.
W2: Along the same line, rediscovering existing conjectures does not necessarily show that the PatternMiner system has worked. Unfortunately, these existing results may have been seen by the agent in pre-training or mid-training, confounding the interpretation of rediscovered conjectures. Is there anything the authors can say / demonstrate to show that PatternMiner is responsible for these rediscoveries rather than it being driven by the LLM’s prior knowledge of the existing conjecture?
W3: PatternMiner is very poorly described and definitely is not reproducible at the moment. Section 3 provides a short and vague description of how the system works, while the Appendices omit any concrete details. I would like to know more about the prompts used, the tools used, and the limits of the kinds of conjectures the system can handle. Furthermore, this is a serious weakness as no code has been submitted. Therefore, from the paper, the appendices, and the lack of code, there is no detailed explanation of the method.
Questions for authors
Q1: Do the authors have any thoughts about applying recent work on using LLM agents for mechanistic interpretability in PatternMiner? For example, [A] claims to have found that most neurons in even very large neural networks have a single job, so very deep networks may be more interpretable than previously thought.
[A] Cammarata et al. 2026. https://already-clean.goodfire.com/
2 comments
Thank you for your response. Unfortunately, given how close we are to the discussion deadline, I have not had sufficient time to carefully evaluate the new material. I will take the response into account when finalising my assessment, but I can not provide further feedback within the discussion period.
Thank you for the thoughtful review and for recognizing conjecture generation as an important research direction. Your comments helped us clarify the scope of our claims and improve both the presentation and documentation of PatternMiner.
Mathematical value of the results. We agree that a conjecture is not valuable simply because it is new. Our motivation is that, as AI systems become better at proving well-specified statements, finding meaningful statements to investigate also becomes increasingly important. PatternMiner is therefore not designed to generate as many new conjectures as possible. Instead, it packages an end-to-end pipeline documenting how conjectures are proposed, tested, refuted, refined, and sometimes proved.
In the revisions, we specifically distinguish between new conjectures, rediscovered results, refuted conjectures, and general statements that were ultimately proved. The strongest outcomes included not only observations and proposed conjectures, but also general mathematical statements that PatternMiner formulated and proved. In one investigation, PatternMiner proved that a natural family of existing tests will always miss certain cases, regardless of how large the mathematical objects become. In another, it proved two previously proposed conjectures. The remaining investigations produced a more efficient way to calculate a mathematical quantity and extended a known result to a broader family of objects. Complete mathematical statements and proofs are provided in Appendix B.
Regarding the research topic, we selected problems from Stanley’s survey because they are well-recognized research problems with several meaningful directions of investigation. However, we agree that this does not replace an independent evaluation of how interesting the results are to the broader mathematical community.
Rediscovery and possible pretraining exposure. We agree that the LLM may have encountered the known results during training, and we cannot determine retrospectively how much its prior knowledge contributed to their rediscovery. For this reason, we do not use rediscovery as evidence that PatternMiner independently discovered those results.
To provide a more informative comparison, we added direct LLM baselines in Section 5 and Appendix C. The direct agents received the same five broad problems and tools for exact computation, but no trained predictor or PatternMiner workflow. They still made useful progress, although their conclusions generally covered more restricted cases or produced less efficient constructions. This helps show what the agents could accomplish without the small-model workflow, although it does not rule out the influence of pretraining.
System description and reproducibility. We agree that the original description was too vague. At a high level, PatternMiner is a markdown-based workflow for a coding agent rather than a single trained model or fixed orchestration program. The workflow contains five prompts, four record templates, and three method guides. The prompts guide the agent through the overall research process, experiment design, model training and interpretation, and conjecture development. The record templates preserve the project’s current state across sessions and document human interventions. The method guides provide additional instructions for model selection, interpretation, and reviewing relevant literature. Section 3 now summarizes this structure and explains each step in more detail. In addition, the anonymous repository provides the exact prompts, templates, guides, experiment records, and code. The models and budgets used for the new raw LLM baselines and controlled interpretation ablation are documented separately.
LLM agents for mechanistic interpretability. PatternMiner already uses an LLM agent to investigate how a trained model arrives at its predictions. In one of our studies, the agent tested how the model’s answers changed when specific internal components were modified. It then used those changes as evidence for constructing a mathematical rule.
We agree that applying similar methods to larger models is a promising direction. We focused on small models because it is easier to inspect their internal behavior, test whether an explanation is accurate, and connect what the model has learned to a precise mathematical statement. It is also immensely less costly than larger models, which is of high value to the open-source academic community.