- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark of ten open/research problems from extremal combinatorics. The benchmark includes a lower and upper bound for each problem and includes machine-checkable verifiers (Lean functions) to check if submissions provide a tighter bound. The problems all fix an integer function lacking a known closed form, such as diagonal Ramsey number or the largest cap set.
They provide evaluation results of agents using Claude Opus 5 that each have at most one hour. They also involve Fable 5.1 and Opus 5.5 on some of the problems. The agents improve upon the baselines stored in the benchmark, but none of these results actually improve on the best results from the published literature.
Critical strengths
It is nice that the models are rigorously evaluated via Lean proofs and they implement some anti-cheat gates that appear to work quite well (e.g., the Golomb trace), making evaluation quite trustable. The paper is also well-written.
Critical weaknesses
-
The set of problems is very narrow. There are only ten problems in one field, and each include only one integer parameter. Furthermore, it seems like the optimal results for the Golomb problem at is already known by exhaustive search, which isn't actually achievable in this budget.
-
There is no evidence of agents showing mathematical discovery. It also seems like the evaluation procedure changed during runs (different prompt between runs and a rule about external sources).
-
The agents are not given that much time to complete the problems. Phase one only permits one hours, and phase two gives two hours. It seems like many other recent results on mathematical discovery with agents allow substantially more time to be spent on the problems.
-
Only Claude models are evaluated. The evaluation results and analysis are very shallow as a result. More models should've been included, especially the most recent GPT models which have made the most progress on research math. Analysis of open-source models would also be nice to include (but not necessary).
Questions for authors
-
What tools are given to the agents in the reasoning process? Web search is allowed, but how come they aren't able to immediately retrieve the best result from the literature?
-
Why were only ten problems included? How many of these problems overlap with other benchmarks?
-
How were the included baseline bounds chosen? They are not the frontier results from the literature, so why were they chosen this way?
-
Why were Claude models the only ones evaluated?
3 comments
Thanks for this reply. I don't find these responses particularly convincing, though. For example, I don't know what "we wanted a baseline from one agent setup and used the one we had" is referring to in the point about only using Claude models. I feel if there was no design reason, then more agents should have been evaluated other than just Claude. Similarly, why was Astra specifically used as the consultant and not Claude? And why did you not evaluate Astra within the pi harness? Many of these choices feel arbitrary and are unjustified. And what does this remark mean: "One point on the summary: phase two covered all 20 records, Fable 5.1 on 19 and Opus 5.5 on one."? Why was Opus given only one problem to work on?
It's also unclear why these specific time limits are chosen. The explanation "The time limits are settings of our runs; the benchmark sets none, and a submission is judged by the record it proves. We did not run longer than this." doesn't say actually explain anything about these design choices which are quite important. In general, I didn't get much clarification about the main points I had, so I will keep my score.
Second Response to Reviewer S4Le
Thank you for the follow-up. We answer each question below and give the reason for each choice.
Claude models. When the gates were finished we had one agent setup ready, Claude Code with the tools listed in App. C. We used it for the baseline on all twenty records and, with two newer Claude models in the same harness, for the second phase. We agree that more model families make the evaluation more informative. The revision therefore adds a third phase with a second harness and a second family, the pi coding agent with GPT-6.1, on ten records (Sec. 5, Table 4).
GPT-6 Astra. In phase two we added one tool, with which an agent could ask a model of another family for a suggestion. With Claude agents this was a GPT model, GPT-6 Astra. Astra answered single queries through this tool and never ran as an agent, so it had no harness. In phase three the agent itself is a GPT model, GPT-6.1, released after phase two.
Opus 5.5 on one record. Our access to Fable 5.1 had a usage limit, which was reached after 19 records. Opus 5.5 ran the twentieth, so that the phase covered all records. The paper draws no comparison between the two models.
Time limits. We chose the limits to fit our budget: one hour per record in phase one, 18.9 hours in all, and a stop notice after 80 minutes in phases two and three. The prompt of phases two and three set no time limit, and an agent ran in one session until it stopped. App. C lists the few early runs with a different setting. In phase three the limit rarely decided the outcome: eight of the ten agents stopped by themselves before the notice (Table 4). These are baseline runs of limited length. We make no claim about what longer runs would prove.
What the runs are for. The paper does not compare models. It proposes a way to measure progress on open problems by records, and the runs show that the records move under verification and that each phase starts from the records of the previous one. The choices above decide which agents produced the current records, not what the benchmark measures.
Response to Reviewer S4Le
Thank you for finding the evaluation trustworthy and the paper well written. Line numbers refer to the submitted PDF, section numbers to the revised PDF (changes in blue).
What the paper claims. We claim no mathematical discovery; the paper reports that no record has a growth rate beyond the literature (ll. 28–30, 457–468). Its contribution is a benchmark that measures progress on open problems by records that stay usable after a success, with a baseline that shows the records moving under verification. In three phases, agents based on four models had 82, 60 and 29 submissions accepted, each phase from the records of the previous one; phase two improved 19 of the 20 records, phase three 9 of the 10 it ran on.
Q1. Tools and retrieval. The agents had a shell, file tools and web search; in phase two they could also consult GPT-6 Astra (App. C). They do find the literature. A published bound is rarely usable as a record as it stands: it is often asymptotic, with unspecified constants, or it rests on a computer search (ll. 139–143). The agent must turn it into a computable function of n and prove it in Lean. That is the work the benchmark credits.
Golomb rulers. The optimal rulers are in the record: the upper record equals the optimal length at every n≤28 and the best known length at every n≤100, each ruler checked by the kernel (ll. 448–450). The budget excludes only the proof that no shorter ruler exists, which came from an exhaustive search and has no known short certificate (ll. 463–468). The budget is meant to make proofs argue instead of letting the kernel search (ll. 214–216).
Q4. Only Claude models. We had no design reason: we wanted a baseline from one agent setup and used the one we had, and the gates check only the submitted files. A non-Claude model was in the loop in phase two, GPT-6 Astra as a consultant, which suggested the counting argument behind the Ruzsa–Szemerédi upper record (Sec. 5). The revision adds a third phase with agents based on GPT-6.1 in a different harness, the pi coding agent, from the records of phase two, with the phase-two prompt (the consultant replaced by a web-search command) and the same gates (Sec. 5, App. C). Any system, including one built on an open-weight model, can be run the same way. One point on the summary: phase two covered all 20 records, Fable 5.1 on 19 and Opus 5.5 on one.
Time per run. The time limits are settings of our runs; the benchmark sets none, and a submission is judged by the record it proves. We did not run longer than this. In phase three, eight of the ten agents stopped on their own before the stop notice at 80 minutes (Table 4).
Changes of procedure. The paper reports them (ll. 416–418), and App. C states which version of the prompt each run used and how the versions differ. The rule on outside sources concerns the licenses and citation of copied code and data; neither change affects what a submission must prove. All records were re-checked under the final gates, and phase three used one prompt throughout. The runs are a baseline and a test of resilience, not a controlled comparison of models, as the paper says.
Q2. Scope and overlap. One integer parameter makes a record a function defined at every n, so that a new record can be proved at least as good everywhere (Sec. 2). Ten problems are admitted so far; the list is open, and Sec. 4 states the criteria. In our check of the public collections, no benchmark or archive poses any of the ten functions as an improvable bound at every n. Seven of the ten are subjects of Erdős problems and two more in a related form (Sec. 4), and Formal Conjectures has asymptotic statements on nine; single values such as R(5,5), R(6,6), S(6) and the 512-cap at n=8 appear in Formal Conjectures, HorizonMath, FrontierMath: Open Problems and FunSearch, and Leaps in Bounds keeps single-n values for five of the ten.
Q3. Initial records. We chose simple initial records on purpose (App. A): they must be computable and proved in Lean, and quick to formalize. The best published bounds rarely meet this; Mathlib's stronger bound for Roth numbers, for example, is noncomputable. Formalizing them is part of the work that the benchmark credits. The current records, in contrast, were not chosen by us. The agents proposed and proved them over the three phases, the gates checked each one before we merged it, and they are now the records that the next submission must beat.