acceptodds
Under review as a conference paper at ICLR 2027

AI Discovery in Combinatorics: From Problems to Solutions and Back to Bigger Problems

Abstract

We introduce Bench, a benchmark of 29 open problems in algebraic combinatorics. These problems ask for a simple combinatorial algorithm that explains given polynomial coefficients or identities. To help AI agents search for such algorithms, we provide a scaffold for playing with combinatorial objects, countering cheating attempts, and testing candidate solutions. To explore how far frontier AI systems can go on these problems, we collaborated with OpenAI, whose researchers obtained proposed solutions to several of them. This highlights bigger meta-problems: is AI discovery at scale ethical? How should these results be released? How should proof digestion be organized? We discuss these questions and give our views.

Primary area:
datasets and benchmarks
Keywords:
A4Math, Algebraic Combinatorics, Frontier Model Evaluation, Mathematical Discovery, AI Ethics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hkw4OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark of (mostly) open problems in algebraic combinatorics called qtqtBench, since problems are drawn from the subfield of q,tq,t-combinatorics, which studies counting problems that arise from parameterized families of polynomials. The tasks include qq-statistic, tt-statistic, and bijection discovery, representing combinatorial interpretations of qq polynomials and/or q,tq,t polynomials.. Agents are provided a harness for exploration and evaluation, which includes a numerical evaluation, a check for cheating attempts, a resource check, and an LLM judge. While GPT-6 Astra (High) did not produce any solutions that passed the entire evaluation pipeline, collaboration with OpenAI using an internal model provided 20 proposed solutions, 10 of which the authors consider to be genuine solutions.

Critical strengths
  • This paper provides a timely benchmark to assess frontier model capability in mathematics. The combinatorics setting provides an especially well-suited set of problems for benchmarking, and the fact that GPT-6 Astra could not solve any problem satisfactorily (and an OpenAI internal model achieves "real" solutions for only 10 of the 29 open problems) demonstrate that qtqtBench remains unsaturated for the time being.

  • The agent scaffold is also a useful contribution, especially considering Brown et al. (2025) also found cheating attempts in their evaluation of agents in an evolutionary harness. Detecting these types of failure modes in the more general setting of general LLM agents is a valuable extension.

Critical weaknesses

As the authors admit in their reproducibility statement, reruns of the baseline evaluation with GPT-6 Astra may give different responses, and runs of an internal OpenAI model in collaboration with OpenAI are totally unreproducible. In particular, this applies both to the solver models and to the LLM judge. This limitation is essentially impossible to overcome for frontier model evaluations, so I do not weigh it against the paper very much, but I think it would be remiss not to restate it here.

Questions for authors

The authors describe their intent to release the AI-generated solutions to researchers who are interested in the work of proof digestion, end they encourage interested authors to coordinate openly. I am curious to know if the authors intend to steward this process in any capacity, what their criteria will be to release the solutions to interested parties, and whether the initial proposed solutions will be made public once the digestion work commences. While I broadly agree with the decision not to release a wave of AI-generated preprints, I also have some reservations about the alternative. Although I do not believe this is the authors' intention, this approach may in some respect put the authors in a position to choose which human mathematicians are allowed to claim credit for digesting a particular solution.

2 comments

We thank the reviewer for the careful reading, for the appreciation of our work, and for the questions on Section 6.

Stewarding and criteria. Yes, we are coordinating the digestion process, and we are sharing the PDFs with members of the community who are interested in the digestion of OpenAI submissions. So far, we have not received requests for the PDFs from people outside the combinatorics community. If this happens, we will try to understand whether there is a real interest in digesting the result.

Public release. No. The original PDFs stay shared with the researchers who are interested in digesting them, and we will not make them public, even once the digestion has started. What becomes public is the digested result, which its authors publish stating that it builds on an AI-generated discovery (Section 6.3). We have made this policy explicit in Section 6.2.

We understand the reviewer's reservation on this release method. This approach has its own drawbacks, as it puts us in a position of power, but it seems to us the best solution, since it is the one that pollutes the community the least.

Thank you for your response. After reading the other reviewers' concerns and the authors' responses, I maintain a positive opinion of this work. In particular, I think the new case study is a good addition.

In my view, the main value of this contribution lies in the benchmark, which has not been saturated and which I find to be a collection of interesting problems. Perhaps this paper raises a question of venue fit at ICLR, but I personally would prefer to see it included.

Reviewer_mHWaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a set of 29 open problems in algebraic combinatorics as a benchmark for AI discovery of deterministic rules for some mathematical phenomena. It goes on to define a scaffold for testing a model on combinatorial objects, including an LLM judge for countering cheating on the benchmark. Candidate solutions using an internal OpenAI model are generated and incompletely evaluated. The authors conclude with a philosophical reflection on AI use in mathematics.

Critical strengths
  1. The mathematical setting of the paper is interesting, in terms of setting up a concrete domain for a model to explore with a well-defined submission for testing, as well as some initial screenings for candidate solutions.
Critical weaknesses
  1. The main paper gives very limited insight on the solutions that were attained within the framework/on the benchmark problems, which makes it difficult to understand the success of the proposed framework for evaluating models on the broader goal of discovering math that explains patterns.

    a. The highlighted OpenAI solutions are not shared in the paper or supplement and the status of many of the solutions is in-progress, which makes the evaluation result unclear. Adding a case study of some solutions and how it relates to the broader goals would make the manuscript stronger.

    b. The method for validating the LLM judge for cheating answers has limited adversarial difficulty, which puts the scalability of the approach in doubt. This weakness is mentioned by the authors when describing the method of the adversarial examples as simple. Furthermore, the same model family is used to generate the adversarial examples as to evaluate them. In this case it would be a notable blind spot of the approach if the model recognizes only the adversarial tricks that it was able to generate but not other independently generated adversarial solutions.

    c. The comparison of the public and private models is limited. There is little quantification of the failure modes or description of progress on the problems before failure.

    d. The automated evaluation utility could be better explained in the case of no code solutions, especially considering that the private model had some solutions that were only writeups.

  2. Although the reflection on AI use in math in section 6 is appreciated, the discussion is not unique to the given study; many similar sentiments have already been shared in various public blog posts. Furthermore, it is premature relative to the study given the lack of insight into the OpenAI solutions.

Questions for authors
  1. Was there anything particularly insightful or novel from the OpenAI solutions? Is it clear whether or not there is something insightful, or are these solutions very opaque? How could these outputs be improved?

  2. Towards the broader goal of an AI that can discover math that explains data, are there other tasks outside this format of combinatorial discovery that could be included in a broader benchmark?

5 comments

(1/2)

We thank the reviewer for the detailed review, for finding the mathematical setting of the benchmark interesting, and for the concrete suggestions, which helped us improve the paper.

(a) We agree that a case study would make the paper much stronger, and we thank the reviewer for pointing this out. We have therefore added a case study of a solution whose digestion we are working on ourselves: problem 5 (Appendix C). There, OpenAI's model found a pattern that, to our knowledge, is new and that enables an elegant solution: a decomposition of each polyomino along its bounce path, which gives an explicit involution exchanging mm and nn, and which also gives a new, manifestly symmetric formula (in the indices m,nm,n) for the q,tq,t-Narayana polynomials. We have described the solution there and shown the insight found by the model.

(b) The highlighted weakness is valid: our test shows that the judge model can spot the cheating attempts that the model itself can produce, and we did not study what happens when the judge model is different from, or weaker than, the solver model. Since the semantic check cannot be automated without an LLM judge, and a weaker judge may miss tricks that a stronger solver can produce, we use a judge model as capable as the solver. This is the setting of all our experiments, where GPT-6 Astra is both solver and judge, and of our test in Appendix A, where the judge, even with medium reasoning effort, flagged all cheating attempts produced by the same model. The weakness therefore does not arise in our experiments. We recommend this choice as part of the evaluation protocol for other solvers.

(c) We thank the reviewer for raising this point. We have added a failure-mode analysis of the GPT-6 Astra baseline (Appendix D). We read all 29 two-hour runs in full (messages, tool calls, submissions and notes) and reconstructed which ideas the model tried and how it spent its time. The runs show what we call "low-entropy thinking" in several forms, for example: in 24 of the 29 runs the first candidate is a published construction close to the problem, and the same few constructions recur in runs that never saw each other; when a construction fails, the model patches it, keeps it as a fallback or proves side results about it; and most new kinds of candidates get a single test before being dropped. We do not have access to the trajectories of OpenAI's internal models, so this analysis covers only the public model. In our experience, human combinatorialists instead try a wider variety of approaches.

(d) The automated evaluation applies only to code: a write-up cannot be run through the checker or the judge, so it is not a solution under the benchmark protocol. All write-up-only proposals come from OpenAI's internal models, which OpenAI ran with its own inference scaffold: we do not control that process, and we do not know why these proposals came without code. We report them for completeness and assessed them manually. Several of them use the Garsia–Milne involution principle, an approach that some combinatorialists avoid and that, in our setting, makes the functions too slow for the checker's time limits; this may be why no code was provided. We marked most of these proposals with a question mark because the value of the model's contribution is not yet clear: several experts in algebraic combinatorics are evaluating their mathematical value, and there is no consensus yet. We have made this explicit in Appendix B.

(2/2)

Section 6. It is true that there are now many opinions and blog posts on very similar topics, and we cite some of them. Our section, however, takes a different route from the one recommended by the AGMAI advisory group in its guidelines of 29 September 2026 (https://agmai.org/general-sep29/), which we now cite in Section 6: we did not immediately publish the PDFs that OpenAI gave us. The reviewer also finds the section premature, given the lack of insight into the OpenAI solutions. We respectfully disagree: it is already clear that the internal models produced genuine results on 10 of the 29 open problems, and that several of them are new, such as the counterexamples to three positivity conjectures and the construction for problem 5 described in the new case study (see (a)). How, where and to what extent these results are new still has to be understood in detail, and this is exactly the work of proof digestion that the section discusses. But the content is there, and we cannot ignore the remarkable success of the internal models: this is why we believe the questions of Section 6 have to be discussed now. For these reasons, we believe the last section is valuable: it is a statement that we need to make to the algebraic combinatorics community, and to the mathematical community more generally, and it argues why we are doing something different from what the AGMAI board recommends to OpenAI.

Q1. The proposed solutions marked with a tick are usually insightful: they spot a new, simple pattern that enables the resolution of the problem. As the reviewer suggests, we have added one of them to the appendix as an example (see (a)). For the proposals marked with a question mark, it is not yet clear how insightful they are, and several experts in the field are working to understand the arguments and distill the new ideas, if any. All of the outputs, even the insightful ones, are badly written and badly presented. For this reason, we believe they need a thorough study, with the goal of understanding and digesting the results, presenting them according to the best mathematical practice, and proposing new conjectures and insights that could guide the future of the field.

Q2. Yes. Outside algebraic combinatorics there are problems similar to ours, for example the discovery of conserved quantities in dynamical systems and of identities in number theory.

In dynamical systems, a task could give an ordinary differential equation x˙=f(x)\dot{x}=f(x) together with some numerical trajectories, and ask the model to find a nonconstant conserved quantity I(x)I(x). A candidate would first be tested on unseen initial conditions, and then validated by proving that ∇I(x)⋅f(x)=0\nabla I(x)\cdot f(x)=0 on the given domain. An output of this kind explains a geometric constraint on the evolution of the system: the trajectories stay on the level sets of II. There are already precedents for learning conservation laws from trajectory data, such as AI Poincaré (Liu and Tegmark, 2021).

In number theory, a task could give high-precision numerical values of constants or special functions, and ask for a compact series or continued-fraction identity with explicit convergence conditions. The Ramanujan Machine (Raayoni et al., 2021) already produces conjectures of this kind from numerical data. Here further numerical evaluations would screen the candidates, and a proof would establish the identity.

We hope that our responses and the revised version address the reviewer's concerns. In particular, as the reviewer suggested, we added a case study of one of OpenAI's solutions (Appendix C) and a failure-mode analysis of the baseline runs (Appendix D). We would be happy to clarify anything further before the end of the discussion period.

Reviewer_ebdkOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark for algebraic combinatorics, focusing on open problems involving finding q-statistic, t-statistic, and bijective arguments. The benchmark consists of a few dozens problems, some of which have been tried and solved by some OpenAI internal models.

While the paper provides an interesting addition to math open problems benchmarks, in my opinion, the contributions of the paper do not meet the bar for the ICLR main conference. Therefore, I recommend rejecting.

Critical strengths

The benchmark picks problems that target a specific capability of LLM, i.e. the capability to generalize initial data into a general conjecture. The problems seem genuinely hard, as evident from the fact that OpenAI models have failed to tackle several of them. Therefore, the benchmark serves well as a capability test for frontier LLM models.

The benchmark is a nice addition to the existing literature, which includes benchmarks of open math problems from different sub-fields of pure mathematics.

Critical weaknesses

In my opinion, the contributions of this paper are not significant enough to appear in the ICLR main conference. According to a summary about ICLR in the conference homepage, the conference is meant for "cutting-edge research on all aspects of deep learning," and I think the paper fails to meet the bar of "cutting-edge research." While the problems in the benchmark are interesting, one can pick equally interesting problems in other sub-fields of mathematics and compile a similar benchmark. The evaluation pipeline is also not very novel - the idea of using automatic checker, then LLM judge, then human judgement is probably one of the first approaches one would consider when designing such a benchmark.

Questions for authors

Lines 308-311: This is more like a suggestion. Why don't you prompt the LLM solver to attempt to give a natural explanation for the statistics it finds?

Lines 394-395: Can you elaborate more on "their tendency towards low-entropy thinking being the main capability bottleneck"?

4 comments

We thank the reviewer for finding the paper clear and sound, and the benchmark a good test of the capability to generalize data into a conjecture.

Significance and novelty. We agree that the evaluation pipeline is not novel, but we believe it is the most reasonable one for these problems. Models cheat, so numerical checks are not sufficient. In discussions with experts we found no formal criterion for what makes a construction natural, so we need a semantic judge and expert review.

The novelty of our work is in the selection of the problems, which are well suited to test frontier models for several reasons.

  • They were open when we built the benchmark, so no solution could be in a training corpus, and any correct one signals genuine discovery.
  • Fit to the data is checked exactly and cheaply by computation.
  • Unlike in benchmarks based on improving bounds, there is no partial contribution, so we measure discovery rather than optimization.
  • Solving them requires long-horizon reasoning and creative exploration. One has to form hypotheses from data, test them, and abandon directions that cannot lead to an exact solution.
  • They matter to mathematicians, and some have been open for decades.

It is true that equally interesting problems exist in other fields, but interest alone does not make a problem suitable. In most fields the solution of an open problem is a proof, which needs expert or formal verification. Here a natural rule that fits the data is already a substantial contribution. We chose q,tq,t-combinatorics because it produces problems of this type systematically.

Why this matters for ICLR. Our experiments on these problems give three results about frontier models. First, GPT-6 Astra (High), with two hours per problem, produced no solution that passed both the checker and the LLM judge. OpenAI's internal models returned proposals for 21 of the 29 open problems, and we consider 10 of them genuine, including counterexamples to 3 positivity conjectures. Hence the benchmark is neither saturated nor out of reach, and it can track this capability as it reaches public models. Second, the model knows what cheating is and cheats anyway. As a judge, GPT-6 Astra (Medium) correctly classified all 51 legitimate and all 51 cheating submissions generated by GPT-6 Astra (Appendix A). As a solver, the same model frequently tried to game the verifier once a problem seemed too hard. Its cheating is therefore an alignment problem. At least 4 of the 21 proposals of OpenAI's internal models also rely on cheating. Third, since the correctness check is cheap, these problems could provide training rewards for discovery agents. However, our results show that numerical verification alone would reward cheating, so such environments need defenses like ours.

Q1 (lines 308–311). We thank the reviewer for the suggestion and we highlight that in our runs we already do this. We explain carefully to both the solver and the judge what counts as a natural solution and what counts as a cheating attempt. Together with its code, the solver must submit a description of the rule and a derivation that justifies each of its choices without using the target data, and in our baseline runs the judge reads both. However, a natural-sounding explanation does not make a rule natural, and both prompts say so explicitly: the judge decides on what the code actually does. So the solver knows what counts as natural, and has to explain why its rule is, but it often cheats anyway.

Q2 (lines 394–395). By "low-entropy thinking" we mean that the model's ideas come from a small, familiar pool, and that it keeps working on familiar constructions rather than developing new ones. To make this precise we read all 29 GPT-6 Astra runs in full and added a failure-mode analysis (Appendix D).

I thank the authors for their comments. I see the overall positive feeback that the benchmark proposed in the paper is timely, suitable, and important for testing capabilities of frontier AI models.

However, I still maintain my opinion that the benchmark is not a great fit for the main ICLR conference, especially as similar papers (such as HorizonMath, FrontierMath) did not appear in a major main conference in AIML.

Due to my more positive understandings of the paper now, I raise my rating to 2 (Weak Rejection).

We thank the reviewer for engaging with our response and for raising the score. We respect the reviewer's view on venue fit. We only note, for completeness, that mathematics benchmarks close to ours have appeared in the main track of ML conferences, for example the research-level algebraic combinatorics datasets of Chau et al. (ICML 2025) and the FATE benchmark for frontier algebra (ICLR 2026).

I thank the authors for bringing up the papers. I will take them into consideration in discussions with the AC and other reviewers, if any.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.