- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark of (mostly) open problems in algebraic combinatorics called Bench, since problems are drawn from the subfield of -combinatorics, which studies counting problems that arise from parameterized families of polynomials. The tasks include -statistic, -statistic, and bijection discovery, representing combinatorial interpretations of polynomials and/or polynomials.. Agents are provided a harness for exploration and evaluation, which includes a numerical evaluation, a check for cheating attempts, a resource check, and an LLM judge. While GPT-6 Astra (High) did not produce any solutions that passed the entire evaluation pipeline, collaboration with OpenAI using an internal model provided 20 proposed solutions, 10 of which the authors consider to be genuine solutions.
Critical strengths
-
This paper provides a timely benchmark to assess frontier model capability in mathematics. The combinatorics setting provides an especially well-suited set of problems for benchmarking, and the fact that GPT-6 Astra could not solve any problem satisfactorily (and an OpenAI internal model achieves "real" solutions for only 10 of the 29 open problems) demonstrate that Bench remains unsaturated for the time being.
-
The agent scaffold is also a useful contribution, especially considering Brown et al. (2025) also found cheating attempts in their evaluation of agents in an evolutionary harness. Detecting these types of failure modes in the more general setting of general LLM agents is a valuable extension.
Critical weaknesses
As the authors admit in their reproducibility statement, reruns of the baseline evaluation with GPT-6 Astra may give different responses, and runs of an internal OpenAI model in collaboration with OpenAI are totally unreproducible. In particular, this applies both to the solver models and to the LLM judge. This limitation is essentially impossible to overcome for frontier model evaluations, so I do not weigh it against the paper very much, but I think it would be remiss not to restate it here.
Questions for authors
The authors describe their intent to release the AI-generated solutions to researchers who are interested in the work of proof digestion, end they encourage interested authors to coordinate openly. I am curious to know if the authors intend to steward this process in any capacity, what their criteria will be to release the solutions to interested parties, and whether the initial proposed solutions will be made public once the digestion work commences. While I broadly agree with the decision not to release a wave of AI-generated preprints, I also have some reservations about the alternative. Although I do not believe this is the authors' intention, this approach may in some respect put the authors in a position to choose which human mathematicians are allowed to claim credit for digesting a particular solution.
2 comments
Thank you for your response. After reading the other reviewers' concerns and the authors' responses, I maintain a positive opinion of this work. In particular, I think the new case study is a good addition.
In my view, the main value of this contribution lies in the benchmark, which has not been saturated and which I find to be a collection of interesting problems. Perhaps this paper raises a question of venue fit at ICLR, but I personally would prefer to see it included.
We thank the reviewer for the careful reading, for the appreciation of our work, and for the questions on Section 6.
Stewarding and criteria. Yes, we are coordinating the digestion process, and we are sharing the PDFs with members of the community who are interested in the digestion of OpenAI submissions. So far, we have not received requests for the PDFs from people outside the combinatorics community. If this happens, we will try to understand whether there is a real interest in digesting the result.
Public release. No. The original PDFs stay shared with the researchers who are interested in digesting them, and we will not make them public, even once the digestion has started. What becomes public is the digested result, which its authors publish stating that it builds on an AI-generated discovery (Section 6.3). We have made this policy explicit in Section 6.2.
We understand the reviewer's reservation on this release method. This approach has its own drawbacks, as it puts us in a position of power, but it seems to us the best solution, since it is the one that pollutes the community the least.