acceptodds
Under review as a conference paper at ICLR 2027

SqueezeBench: Saturation-resilient evaluation of mathematical capability

Abstract

AI systems are making rapid progress in mathematics, but this progress is hard to measure. Benchmarks of solved problems saturate, and in benchmarks of open problems each problem can be resolved for the first time only once. We present SqueezeBench, a benchmark that is resilient to saturation: a success raises the bar instead of removing the problem. Each problem fixes an integer function whose values are known only for small , and the benchmark keeps a lower and an upper record on it. A record is a computable Lean function with a machine-checked proof that it bounds . A submission proposes a new function and proves that it bounds , that it is no worse than the current record at any , and that it is strictly better at some . Once accepted, it becomes the record that the next submission has to beat. The system thus chooses the bound as well as its proof, and it may improve the record at a single or improve its growth rate. At every , the gap between the two records shows how far the certified bounds are from determining . Every proof is checked by two independent Lean kernels, so no referee needs to check the mathematics. The checks also reject a function that computes by exhaustive search at small , and a maintainer only reads the submitter's report and looks for such a search at larger . We release ten problems from extremal combinatorics with twenty proved initial records, together with the admission tools. As a baseline, agents based on a frontier language model, with at most one hour per record, had 82 submissions accepted and improved every record, without going beyond the published literature. Agents based on two newer models then started from these records, had 60 more submissions accepted and improved 19 of the 20 records again. For eight of the ten problems, at least one record is still asymptotically behind the best published bound.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, benchmark saturation, AI for mathematics, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S4LeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a benchmark of ten open/research problems from extremal combinatorics. The benchmark includes a lower and upper bound for each problem and includes machine-checkable verifiers (Lean functions) to check if submissions provide a tighter bound. The problems all fix an integer function lacking a known closed form, such as diagonal Ramsey number or the largest cap set.

They provide evaluation results of agents using Claude Opus 5 that each have at most one hour. They also involve Fable 5.1 and Opus 5.5 on some of the problems. The agents improve upon the baselines stored in the benchmark, but none of these results actually improve on the best results from the published literature.

Critical strengths

It is nice that the models are rigorously evaluated via Lean proofs and they implement some anti-cheat gates that appear to work quite well (e.g., the Golomb trace), making evaluation quite trustable. The paper is also well-written.

Critical weaknesses
  • The set of problems is very narrow. There are only ten problems in one field, and each include only one integer parameter. Furthermore, it seems like the optimal results for the Golomb problem at n=28n=28 is already known by exhaustive search, which isn't actually achievable in this budget.

  • There is no evidence of agents showing mathematical discovery. It also seems like the evaluation procedure changed during runs (different prompt between runs and a rule about external sources).

  • The agents are not given that much time to complete the problems. Phase one only permits one hours, and phase two gives two hours. It seems like many other recent results on mathematical discovery with agents allow substantially more time to be spent on the problems.

  • Only Claude models are evaluated. The evaluation results and analysis are very shallow as a result. More models should've been included, especially the most recent GPT models which have made the most progress on research math. Analysis of open-source models would also be nice to include (but not necessary).

Questions for authors
  • What tools are given to the agents in the reasoning process? Web search is allowed, but how come they aren't able to immediately retrieve the best result from the literature?

  • Why were only ten problems included? How many of these problems overlap with other benchmarks?

  • How were the included baseline bounds chosen? They are not the frontier results from the literature, so why were they chosen this way?

  • Why were Claude models the only ones evaluated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer S4Le

Thank you for finding the evaluation trustworthy and the paper well written. Line numbers refer to the submitted PDF, section numbers to the revised PDF (changes in blue).

What the paper claims. We claim no mathematical discovery; the paper reports that no record has a growth rate beyond the literature (ll. 28–30, 457–468). Its contribution is a benchmark that measures progress on open problems by records that stay usable after a success, with a baseline that shows the records moving under verification. In three phases, agents based on four models had 82, 60 and 29 submissions accepted, each phase from the records of the previous one; phase two improved 19 of the 20 records, phase three 9 of the 10 it ran on.

Q1. Tools and retrieval. The agents had a shell, file tools and web search; in phase two they could also consult GPT-6 Astra (App. C). They do find the literature. A published bound is rarely usable as a record as it stands: it is often asymptotic, with unspecified constants, or it rests on a computer search (ll. 139–143). The agent must turn it into a computable function of nn and prove it in Lean. That is the work the benchmark credits.

Golomb rulers. The optimal rulers are in the record: the upper record equals the optimal length at every n≤28n\le 28 and the best known length at every n≤100n\le 100, each ruler checked by the kernel (ll. 448–450). The budget excludes only the proof that no shorter ruler exists, which came from an exhaustive search and has no known short certificate (ll. 463–468). The budget is meant to make proofs argue instead of letting the kernel search (ll. 214–216).

Q4. Only Claude models. We had no design reason: we wanted a baseline from one agent setup and used the one we had, and the gates check only the submitted files. A non-Claude model was in the loop in phase two, GPT-6 Astra as a consultant, which suggested the counting argument behind the Ruzsa–Szemerédi upper record (Sec. 5). The revision adds a third phase with agents based on GPT-6.1 in a different harness, the pi coding agent, from the records of phase two, with the phase-two prompt (the consultant replaced by a web-search command) and the same gates (Sec. 5, App. C). Any system, including one built on an open-weight model, can be run the same way. One point on the summary: phase two covered all 20 records, Fable 5.1 on 19 and Opus 5.5 on one.

Time per run. The time limits are settings of our runs; the benchmark sets none, and a submission is judged by the record it proves. We did not run longer than this. In phase three, eight of the ten agents stopped on their own before the stop notice at 80 minutes (Table 4).

Changes of procedure. The paper reports them (ll. 416–418), and App. C states which version of the prompt each run used and how the versions differ. The rule on outside sources concerns the licenses and citation of copied code and data; neither change affects what a submission must prove. All records were re-checked under the final gates, and phase three used one prompt throughout. The runs are a baseline and a test of resilience, not a controlled comparison of models, as the paper says.

Q2. Scope and overlap. One integer parameter makes a record a function defined at every nn, so that a new record can be proved at least as good everywhere (Sec. 2). Ten problems are admitted so far; the list is open, and Sec. 4 states the criteria. In our check of the public collections, no benchmark or archive poses any of the ten functions as an improvable bound at every nn. Seven of the ten are subjects of Erdős problems and two more in a related form (Sec. 4), and Formal Conjectures has asymptotic statements on nine; single values such as R(5,5)R(5,5), R(6,6)R(6,6), S(6)S(6) and the 512-cap at n=8n=8 appear in Formal Conjectures, HorizonMath, FrontierMath: Open Problems and FunSearch, and Leaps in Bounds keeps single-nn values for five of the ten.

Q3. Initial records. We chose simple initial records on purpose (App. A): they must be computable and proved in Lean, and quick to formalize. The best published bounds rarely meet this; Mathlib's stronger bound for Roth numbers, for example, is noncomputable. Formalizing them is part of the work that the benchmark credits. The current records, in contrast, were not chosen by us. The agents proposed and proved them over the three phases, the gates checked each one before we merged it, and they are now the records that the next submission must beat.

Thanks for this reply. I don't find these responses particularly convincing, though. For example, I don't know what "we wanted a baseline from one agent setup and used the one we had" is referring to in the point about only using Claude models. I feel if there was no design reason, then more agents should have been evaluated other than just Claude. Similarly, why was Astra specifically used as the consultant and not Claude? And why did you not evaluate Astra within the pi harness? Many of these choices feel arbitrary and are unjustified. And what does this remark mean: "One point on the summary: phase two covered all 20 records, Fable 5.1 on 19 and Opus 5.5 on one."? Why was Opus given only one problem to work on?

It's also unclear why these specific time limits are chosen. The explanation "The time limits are settings of our runs; the benchmark sets none, and a submission is judged by the record it proves. We did not run longer than this." doesn't say actually explain anything about these design choices which are quite important. In general, I didn't get much clarification about the main points I had, so I will keep my score.

AuthorsOpenReview ↗

Second Response to Reviewer S4Le

Thank you for the follow-up. We answer each question below and give the reason for each choice.

Claude models. When the gates were finished we had one agent setup ready, Claude Code with the tools listed in App. C. We used it for the baseline on all twenty records and, with two newer Claude models in the same harness, for the second phase. We agree that more model families make the evaluation more informative. The revision therefore adds a third phase with a second harness and a second family, the pi coding agent with GPT-6.1, on ten records (Sec. 5, Table 4).

GPT-6 Astra. In phase two we added one tool, with which an agent could ask a model of another family for a suggestion. With Claude agents this was a GPT model, GPT-6 Astra. Astra answered single queries through this tool and never ran as an agent, so it had no harness. In phase three the agent itself is a GPT model, GPT-6.1, released after phase two.

Opus 5.5 on one record. Our access to Fable 5.1 had a usage limit, which was reached after 19 records. Opus 5.5 ran the twentieth, so that the phase covered all records. The paper draws no comparison between the two models.

Time limits. We chose the limits to fit our budget: one hour per record in phase one, 18.9 hours in all, and a stop notice after 80 minutes in phases two and three. The prompt of phases two and three set no time limit, and an agent ran in one session until it stopped. App. C lists the few early runs with a different setting. In phase three the limit rarely decided the outcome: eight of the ten agents stopped by themselves before the notice (Table 4). These are baseline runs of limited length. We make no claim about what longer runs would prove.

What the runs are for. The paper does not compare models. It proposes a way to measure progress on open problems by records, and the runs show that the records move under verification and that each phase starts from the records of the previous one. The choices above decide which agents produced the current records, not what the benchmark measures.

Reviewer_JRVsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is generally clear, although some mathematical notation required additional clarification for me. I used AI assistance to clarify the content and polish this review. I take responsibility for the assessment below.

SqueezeBench evaluates mathematical capability through progressively tighter bounds on open, parameterised problems of the form L(n)≤f(n)≤U(n)L(n)\leq f(n)\leq U(n). A system proposes an improved lower or upper bound and supplies Lean proofs establishing its correctness, showing that it never weakens the previous record, and demonstrating a strict improvement for at least one parameter value. Once accepted, the improvement becomes the target for subsequent submissions.

The current benchmark contains ten problem families from extremal combinatorics, including cap sets, Ramsey numbers and Golomb rulers. The framework allows additional problems to be incorporated.

Critical strengths
  • The strongest contribution is the combination of persistent mathematical records, bounds covering every parameter value, and formal guarantees that accepted updates improve the record without regression. I find this combination moderately novel and interesting.

  • The benchmark recognises intermediate mathematical progress, including improvements at individual parameter values and improvements to asymptotic bounds.

  • The formalised problem definitions, initial records and verification infrastructure provide a useful community resource. This work reduces the effort required for researchers to construct and evaluate agents on these tasks.

Critical weaknesses
  • The current coverage is limited: the benchmark contains ten problem families, all within extremal combinatorics. This makes it difficult to assess how broadly the framework can evaluate mathematical capability.

  • I would appreciate a stronger justification of the problem selection, particularly its diversity, representativeness and expected scope for continued improvement. A larger collection would be valuable, but the choice of problems matters alongside their number.

Questions for authors
  • Could the benchmark incorporate suitable problems from other public collections of open mathematical problems, particularly from areas beyond extremal combinatorics?

  • What is the main bottleneck in adding a problem: selecting an appropriate target, formalising its definition, proving initial bounds, or implementing the surrounding infrastructure? How much of this process can coding agents automate?

My recommendation is weak accept because I find the framework useful and its central design interesting. Broader mathematical coverage, or stronger evidence that the current selection provides a sufficiently representative evaluation, would increase my confidence in a clear accept.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JRVs

Thank you for the assessment and for the questions on scope and selection. Changes are blue in the revised PDF, and section numbers refer to it.

Coverage. Ten problems from one area cannot show how broadly the framework evaluates mathematics, and Sec. 6 says so. What the current selection supports is an evaluation within extremal combinatorics. The framework applies to any problem that meets four criteria (Sec. 4): the answer is an integer function of one parameter with no known closed form; a few exact small values are known; the literature bounds it from both sides, so that both records can move; and Mathlib has what the definition and two simple initial bounds need.

Diversity. The ten come from three subareas: additive combinatorics, extremal graph theory and Ramsey theory. Their records use different methods (Table 2, App. D): algebraic constructions over finite fields, counting arguments, the probabilistic method, the polynomial method of Ellenberg and Gijswijt, and explicit objects found by computer search and checked by the kernel.

Room for improvement. For eight of the ten problems at least one record is still asymptotically behind the best published bound (Table 2), and where a record has reached it, f(n)f(n) stays unknown for large nn.

Q1. Problems from other collections. Yes. Nine of our ten functions are already subjects of problems on erdosproblems.com, seven directly, for example #86 for C4C_4-free subgraphs of the hypercube and #573 for graphs without cycles of length 3 or 4, and two in a related form; Formal Conjectures has asymptotic statements on nine. SqueezeBench poses the same functions as records at every nn. Beyond extremal combinatorics we have screened candidates from coding theory, binary codes A(n,4)A(n,4) and covering codes K(n,1)K(n,1), and, within it, tournament Ramsey numbers and the coil-in-the-box problem. We name them as candidates and have not decided to add them. A problem from another collection enters by the steps of App. A if it meets the four criteria; for an asymptotic conjecture this means defining its function at every nn and proving a simple initial bound in each direction.

Q2. Bottleneck and automation. In our experience most of the work goes into two steps: screening a candidate against the four criteria, and the Lean definition of ff with its well-posedness proof. The initial records took little work, and the infrastructure is shared: once the definition, the initial records and a configuration file exist, the tool squeeze handles the new problem (App. A). On automation: AI coding tools helped us write the Lean definitions and the initial records, as the AI-use statement says, and the records of Sec. 5 were produced by the agents under evaluation. We did not measure how much an agent can do alone, and we give no effort figure. One step should stay with a human: the check that the Lean definition of ff describes the intended problem. The revision adds a mechanical aid to it: where the two records meet they determine f(n)f(n), and these values are now kernel-checked theorems, 88 points with n≤100n\le 100, all in agreement with the published values.

Evaluation. The revision adds a third phase (Sec. 5, App. C). Agents based on GPT-6.1 ran in a different harness, the pi coding agent, from the records left by phase two, with the phase-two prompt (the consultation of GPT-6 Astra replaced by a web-search command) and the same gates. They had 29 submissions accepted and improved nine of the ten records they ran on, all but the upper record for Schur numbers: the nine records that Table 2 leaves unmarked and, as a test of the setup, the lower record for Golomb rulers. Tables 3 and 4 list every track of the three phases, and App. D sorts the 20 current records by the kind of their main ingredient.

Reviewer_6g1ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SqueezeBench for mathematical capability, designed to resist saturation. Each problem is an integer function f(n) from extremal combinatorics (e.g., Golomb rulers), whose exact values are known only for small n. The benchmark maintains a lower and an upper record on f, and each is a computable Lean 4 function. A submission replaces a record only by proving three obligations: (1) the new function bounds f (valid), (2) the new function is no worse than the current record at every n (improves), and (3) the new function is strictly better at some n (strict). An accepted submission then becomes the new target, and each success raises the bar instead of removing the problem. The gap between records at each n quantifies how far the certified bounds are from determining f(n).

Critical strengths
  1. Problem framing: This paper targets a real and timely pain point of current math benchmarks, which include saturation, answer leakage, and the one-shot nature of open-problem benchmarks. The author also presents a clear design principle that directly addresses the aforementioned problems.
  2. Method design: For the method design, the three obligations (valid, improves, strict) are simple and complete, which matches the claim that records improve monotonically. The author uses an anti-gaming mechanism: the authors addressed situations such as records not referencing f, proofs having a heartbeat budget, and records being cheaply evaluable for n ≤ 100 to exclude exhaustive search.
  3. Comparison to related work: The authors systematically compare the distinction against multiple existing works
  4. Evaluation: The accepted records are kernel-verified and independently re-verified. The authors are also honest about limitations that the two evaluation phases are explicitly not a controlled model comparison.
Critical weaknesses
  1. In Section 1, the paper motivates SqueezeBench by the saturation and answer leakage of existing benchmarks. However, is SqueezeBench itself exposed to leakage? Say, if the accepted records, proofs, and reports are public, and future models could also be trained on these Lean proofs.
  • Since the initial records are weaker than published bounds, how should we distinguish genuine mathematical reasoning ability from retrieval or memorization of existing results and proofs?
  • When comparing models from a fixed historical snapshot, do you plan to prevent subsequently published records and proofs from leaking through training data or web access?
  1. In Section 2, the mechanical check against exhaustive search only covers n≤100n \leq 100; for n≥100n \geq 100 the benchmark relies entirely on a maintainer manually reviewing REPORT.md. However, this step can be easily bypassed: a submitter could hide a search behind if n ≤ 100 then formula else search and describe only the first branch in the report. Gate 6 was also introduced midway through the experiments, which suggests the threat model is still evolving. I would suggest that the authors to specify the concrete criteria maintainers use in this review, or to consider stronger mechanical safeguards such as requiring records to be evaluable at randomly sampled larger n, or restricting the branching structure permitted in Def.lean. Additionally, whether the Lean definition of f matches the intended problem is checked only once, by a maintainer. If that definition is wrong, every subsequent record proves a bound on the wrong object, and the paper offers no mitigation for this risk.

  2. In Section 4, key implementation details are also missing: the actual heartbeat budgets of Gates 4 and 6, the Lean/Mathlib/Comparator versions (which the machine-independence claim depends on), the seed records and formal definitions for nine of the ten problems, and which prompt and rule versions applied to which runs. The appendix covers only adding a problem. I would ask the authors to add the full experiment protocol in the appendix.

  3. In Section 5, the authors only report the empirical results in a selective way. Only the Golomb ruler problem is discussed in detail. Other problems appear only as selected highlights (e.g., the Ruzsa–Szemerédi upper bound, Ellenberg–Gijswijt for cap sets, etc.). For the remaining records. Table 2 only reports the final state after both phases, which does not include per-phase results. The paper could be strengthened by adding per-record, per-phase results with the exact budget.

  4. There is a lack of in-depth analysis to the existing results. For example, reusing existing formalizations, new formalizations of known results, and improvements beyond the literature can suggest different levels of model capability. What capabilities does the benchmark measure?

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6g1R

We followed most suggestions: written review criteria, the full protocol, a table of all runs, a check of the definitions and a sorting of the records. We did not adopt the two mechanical safeguards. Changes are blue in the revised PDF, and section numbers refer to it.

Leakage. For large nn nobody knows f(n)f(n), so there is no answer key. Accepted records and proofs are public on purpose and may enter training data. A submission is accepted only if it beats the current record at some nn, so reproducing a public record earns nothing (Sec. 1).

Reasoning or retrieval. Our initial records are weaker than the published bounds, so the first improvements can come from the literature, and we report this catch-up phase as such. A published bound is not yet a record: it is often asymptotic, with unspecified constants, and a record must be a computable function of nn with a proof that the kernel accepts.

Old snapshots. We do not compare from old snapshots. A controlled comparison starts every system from the current records, where no newer record exists to look up; Sec. 1 now says "current". Web access is a harness setting, and training data cannot be controlled, which is one more reason against old snapshots.

A search hidden above n=100n=100. The reviewer is right about our wording: three of four sentences said that the maintainer reads the report, and no criteria were written down. The design document in the supplement says that the review of Def.lean rejects such a record. We built the reviewer's record on the tutorial problem, a formula for n≤100n\le 100 and an exhaustive search above. It passed all six gates, and its evaluation at n=101n=101 hit the 8 GiB memory limit after 12 seconds. What such a record can do is limited: gate 6 evaluates both records at the witness, so its improvement must sit at some n≤100n\le 100, or the search would run there; its bound is correct; its harm is that nobody can evaluate or improve it above 100 within the limits. The revision corrects the wording (Sec. 2) and writes the criteria down (App. E). The maintainer reads Def.lean, which gate 2 keeps small, and the report, and rejects the submission unless every branch of a case split on nn is an explicit bound of a recognizable kind, no part restates ff or enumerates the objects behind it, and the report describes every part. The reviewer's record is the worked case, and it is rejected under the second criterion.

The two safeguards. Both would reject records that hide no search. Forbidding case splits would reject 42 of the 162 record definitions of the submitted supplement, which branch on a threshold of nn for ordinary reasons, such as a table below it. Evaluation at larger nn under the current limits fails for 5 of the 20 current records somewhere between n=105n=105 and n=400n=400, on memory and not on a search (App. E), so we did not make it a gate. The mechanical check stops at n=100n=100, the review covers the rest, and the range end is a per-problem setting that a maintainer can raise where the current record allows.

Gate 6. From the start, gate 6 evaluated each new record at the witness. The evaluation at every n≤100n\le 100 was added during phase two; every record of phases one and two was then re-checked under the final gates, and phase three ran under them throughout.

The definition of ff. App. B prints the Lean definition of ff for all ten problems. Both records are proved against the same ff, so where they meet they determine f(n)f(n). The revision adds these values as kernel-checked theorems, 88 points with n≤100n\le 100, such as R(4,4)=18R(4,4)=18, and every one agrees with the published value. This is a finite check, not a proof of the definition.

Protocol. App. C gives the heartbeat budgets of gates 4 and 6, the memory and time limits, the versions of Lean, Mathlib, Comparator and nanoda, the tools of each phase and the prompt version of every run. Table 3 lists the initial records.

Per-phase results. Table 3 lists every track in phases one and two, with versions, sessions, minutes and how it ended; Table 4 lists the ten runs of the third phase.

What the benchmark measures. Progress: every accepted version is kept, so the history shows each step. Reach: how far a system moves the records, against the published bounds. Choosing a bound: none is given, so the system must propose a computable function of nn that is valid and beats the record. Proof development and formalization: it must develop a proof that passes the six gates, whose budget limits the checks, not the system's time. On the reviewer's three levels, App. D sorts the 20 current records by main ingredient. 8 are published arguments formalized by an agent. 12 are refinements that the agent presents as its own; we did not judge novelty. One of the 12 also keeps a port of a public Lean formalization, and 11 keep a published growth rate. No growth rate goes beyond the literature; one record may do so at concrete nn.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision and of our replies

We thank the three reviewers. We reply to each review in its own thread. This comment summarizes the revised PDF, in which all changes are marked in blue.

How SqueezeBench differs from other benchmarks. Benchmarks that ask for the proof of a fixed statement, such as PutnamBench or FrontierMath Erdős, score each problem as solved or not solved. A solved problem no longer tells systems apart, so these benchmarks saturate, and they give little or no credit for a better bound or for the earlier work that a final proof builds on (Sec. 1, Sec. 3). HorizonMath does credit a better value, but against a fixed baseline (Table 1).

SqueezeBench keeps two records for each problem, a lower and an upper bound on ff, each a function of nn. An accepted submission becomes the new version of a record. It must bound ff, be at least as good as the previous version at every nn, and beat it at some nn; two independent Lean kernels check the three proofs. Each version is stored with its author, its date and the nn at which it improved the record, so the record history shows every step, and squeeze audit can re-check every version (Sec. 1, Sec. 2). Record archives such as Leaps in Bounds also keep improvable records, as values at single nn (Table 1).

Two uses of the benchmark. The main output is the record history: each accepted version is the record that the next system must beat, so a success makes the benchmark harder. The benchmark also permits a controlled comparison: one starts several systems from the current records with the same budget and compares the records they reach at each nn.

Leakage and retrieval. All ten problems are open beyond small nn, so there is no answer key that could leak. The current records are public, and reproducing one earns nothing. Published bounds can help a system catch up with the literature, and a published bound becomes a record only as a computable function of nn with a proof that the kernel accepts.

Results of computer search. Constructions found by computer search enter the records as explicit objects that the kernel checks, such as the optimal Golomb rulers up to 28 marks. Impossibility proofs by exhaustive search enter only within the computation budget, which is meant to make each proof a mathematical argument; the proof that no shorter ruler with 28 marks exists is far beyond it.

Changes in the revision (in parentheses: the reviewers who asked)

  1. A third evaluation phase (Sec. 5, App. C, Table 4). Agents based on GPT-6.1 ran in the pi coding agent, a different harness, from the records left by phase two, with the phase-two prompt (the consultation of GPT-6 Astra replaced by a web-search command) and the same gates. They had 29 submissions accepted and improved 9 of the 10 records they ran on. Eight of the ten agents stopped on their own before the stop notice at 80 minutes. (S4Le)
  2. A list of every track in every phase, with model, prompt version, accepted versions, minutes and how the run ended (Tables 3 and 4, App. C). (6g1R)
  3. The full protocol: budgets, limits, tool versions and the tools of each phase (App. C). (6g1R)
  4. The Lean definition of ff for all ten problems (App. B), and the values of ff at the 88 points with n≤100n\le 100 where the two records meet, as theorems checked by the kernel. (6g1R)
  5. The review of a submission (Sec. 2, App. E): the maintainer reads the submitted definition Def.lean, as the design document already said, now by written criteria; the worked case is a record that switches to an exhaustive search above n=100n=100. (6g1R)
  6. What the current records consist of, and what the benchmark measures today (Sec. 5, App. D). (6g1R)
  7. The criteria for admitting a problem, and the subareas and methods of the ten problems (Sec. 4). (JRVs, S4Le)
  8. A comparison of systems starts from the current records (Sec. 1), and the benchmark also permits such a controlled comparison (Sec. 6). (6g1R)

Over the three phases, agents based on four models had 171 submissions accepted (82, 60 and 29). For eight of the ten problems, at least one record is still asymptotically behind the best published bound.

AuthorsOpenReview ↗

Additional note on the summary

One addition to the first point above. The mathematician Jordan Ellenberg said about the Erdős unit distance problem that after the answer to the yes-or-no question, "the question of how well can you do remains alive", and that treating the problem as closed is "a very, very meager view" (3Blue1Brown podcast, "Jordan Ellenberg on Being Dispensable", 18 Nov 2026, at 18:45). The records of SqueezeBench are meant to measure exactly that remaining question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.