acceptodds
Under review as a conference paper at ICLR 2027

The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

Abstract

We introduce the Endless Exam, a benchmark spanning fourteen parameterised families of mathematical construction problems, with verifiable scores that distinguish progress before and beyond published mathematical frontiers. Each submitted object is checked automatically for validity and assigned a relative quality score against a published frontier or construction baseline, without capping improvements at 1. The benchmark draws long-term challenges from open mathematical problems and generates larger instances by varying their parameters. Compact certificates allow large constructions to be verified without listing every element. Across nine models evaluated on 69 distinct instances, continuous quality scores distinguish performance even though none of the 30 published-frontier references is surpassed. Size–quality curves show how construction quality changes as problem size increases. We release the generators, verifiers, references, model responses and analysis to support continued measurement before and beyond human frontiers.

Primary area:
datasets and benchmarks
Keywords:
large language models, evaluation, mathematical reasoning, benchmarks, AI for mathematics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rBTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Endless Exam, a benchmark of 69 instances from 14 families of mathematical construction problems. It uses automatic verification, compact certificates for large constructions, and quality scores relative to published frontiers or construction baselines. The evaluation covers nine models with different reasoning efforts and tool settings, and examines how construction quality changes with problem size.

Critical strengths

The model coverage is broad, including commercial and open models, different model sizes, reasoning efforts, and tool settings. This provides a useful comparison of current systems.

The verification and released materials are thorough. The code and saved results are consistent with the main reported results.

Critical weaknesses

The contribution seems incremental relative to existing construction and open-problem benchmarks. The paper mainly combines existing problem families, parameter variations, and relative scoring. The 69 instance evaluation and short size sweeps provide limited support for the much broader “endless” and “superintelligence” framing. Increasing parameters alone does not establish that the resulting tasks will continue to provide useful measurements. Particularly, I think the title of this paper doesn't match the content well since there is limited evidence and discussion on "endless" and no connection with the "toward superintellegence".

The overall score is difficult to interpret. Of the 69 instances, 39 use construction baselines that may be weaker than published results. Averaging the quality ratios also gives some instances disproportionate influence: the length-64 trifference instance accounts for about 71% of Opus’s overall lead over Astra in the tool-assisted evaluation. I am not convinced that this score reliably reflects broad improvements across the suite.

Compared to prior arts, the size of the problem set is small, inconsistent with it's "endless" framing.

Questions for authors

What evidence from the current experiments supports the claim that parameter variation will sustain useful evaluation beyond current models? How do you distinguish increased mathematical difficulty from greater output requirements or limitations of the accepted representations?

What justifies averaging quality ratios across objectives with very different scaling, such as code cardinality and geometric area? Appendix H reports stability when excluding individual families, but do you have additional analysis showing sensitivity to stronger construction baselines or less outlier-sensitive aggregation?

A suggestion beyond questions: I appreciate the authors' effort in presentation and visual design. However, as in a text rich paper, I think figure 1 and those in appendix A are not necessary given the low information density they have. Those figures, although can be great visualization in slides and videos, contains mainly texts that are already described in the main body, and the illustrations don't help a lot here. I suggest the author to create a denser and more detailed main figure as figure 1.

2 comments
AuthorsOpenReview ↗

Response to Reviewer rBTE

Thank you for checking the released code and results. We completed additional scoring and representation analyses using the original responses.

Contribution and scaling evidence. Endless Exam measures construction quality after validity has saturated: Astra has 69/69 valid answers in both tracks, while Overall rises from 91.90 to 143.16 with tools. Compared with correctness-based construction evaluation, this records the quality of valid solutions on research optimisation problems. Compact certificates extend the measurement to submitted codes with over a trillion codewords.

The following Luna high series each have two valid, untruncated answers at every size:

SeriesDimensionsMean relative qualityFinal JSON size
Cap sets, product descriptions8, 10, 120.781, 0.633, 0.508191–297 bytes
Spherical codes, orbit descriptions10, 13, 160.667, 0.516, 0.11159–1,578 bytes

These declines occur with short completed descriptions, without output truncation. Astra's cap-set products instead retain quality 0.970 at dimension 12. The curves distinguish methods that preserve quality from those that fall behind as size grows. The 69 instances are the released evaluation set; the families generate further tasks under the same rules. Scoring beyond reference parity and expansion to larger parameters provide the concrete basis for “Endless Exam.” The title's “toward superintelligence” reflects the intended progression: measuring better constructions on existing instances, then extending evaluation to larger instances as models advance.

Aggregation, references and concentrated gains. A ratio of 1.2 means 20% more codewords or 20% larger minimum triangle area than the instance's reference. Averaging these ratios summarises relative construction quality across instances. The difficulty of achieving those gains depends on the task and reference.

Using 11 independently verified stronger references gives tool scores of 167.69 for Opus and 132.39 for Astra and preserves all 17 configuration ranks; all remaining references stay unchanged. A less outlier-sensitive summary, 100 × (exp(mean(log(1+r))) − 1) with r denoting relative quality, gives 140.33/132.14. These directly test the reference and aggregation choices raised in your question.

Length-64 trifference contributes 26.09 of Opus's 36.90-point lead, reflecting three times as many verified codewords. Opus's largest gains occur in Heilbronn and length-64 trifference, while Astra achieves the highest mean on published-frontier instances.

Representation and presentation. At trifference length 64, the current formats have an exact ceiling of 531,441 codewords, attained by Opus and documented in Appendix D.5. Scoring rewards improvements beyond reference parity within the accepted formats. In the revision, we will put these format limits beside the main description and make Figure 1 more informative by connecting the scoring and short-answer scaling evidence to the benchmark design, reducing repetitive illustrations.

We hope these results provide a clearer basis for reassessing the empirical contribution.

Thank you for answering my question. I still found the design highly heuristic and my concerns on "overclaiming" or putting irrelevant terms (superintellegence...) in the title and abstract unsolved. Given the limited size of this "endless exam", I am not convinced to upgrade my score.

Reviewer_kUeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark called Endless Exam, consisting of 69 instances from fourteen parameterized families of mathematical construction problems. Unlike many traditional benchmarks, it uses uncapped scores to measure model performance, allowing models to achieve higher scores beyond existing references and helping avoid benchmark saturation. By changing parameters such as dimensions and graph degrees, the benchmark can easily generate new problems. The authors evaluate the benchmark on several current frontier models, including GPT-6 Astra and Claude Opus 5.5, under both tool-free and tool-assisted settings. The results show that the benchmark can distinguish model performance and measure how construction quality changes with problem size.

Critical strengths
  1. The idea of using uncapped scores to avoid benchmark saturation is interesting. Also, the selected mathematical problems allow the benchmark to be easily expanded by changing dimensions, graph degrees, and other parameters.

  2. The benchmark supports two answer formats: explicit objects and compact descriptions. This allows models to provide different types of mathematical constructions, including very large ones.

Critical weaknesses
  1. My main concern is the fairness of the overall scoring. Among the 69 problems, 30 use published mathematical frontiers as references, while the other 39 use baselines generated by the authors' baseline algorithms. Improving upon a published mathematical frontier can be much harder than improving upon an algorithm-generated baseline. For example, Astra achieves a higher score than Opus 5.5 on the 30 published-frontier problems (0.998 vs. 0.985). However, Opus 5.5 performs better on the construction-baseline problems (2.43 vs. 1.76). These differences do not necessarily represent the same level of mathematical improvement. Directly averaging these scores may lead to an unfair overall ranking.

  2. In Section 5.3, the authors increase the search budget from 10 to 600 seconds and observe limited improvement for some problems. However, this ablation experiment is not sufficient to establish general search resistance of problems established. The limited improvement may come from the weakness of search algorithms currently using rather than the difficulty of the problems. It would be more convincing to compare with other search methods.

Questions for authors
  1. Can the authors compare the model rankings based on the two reference groups separately to check the rank stability?

  2. The paper reports that Claude Fable 5.1 has worse performance when using high-effort compared to medium-effot, as the responses exhausted the 128K token budget. Could the authors report how many responses were truncated due to this limit for each model, and how much these truncations affect the overall rankings? I am concerned that the benchmark rankings may be influenced too heavily by token efficiency rather than reflecting the models' actual mathematical abilities.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kUeN

Thank you for requesting the group-ranking and truncation analyses. We computed both from the original saved responses.

Reference groups and ranking. The tool-assisted group means are:

Reference groupAstra highLuna highOpus 5.5 high
Published frontiers, 30 instances0.99840.96090.9851
Construction baselines, 39 instances1.76481.37252.4279

Astra leads on published frontiers, while Opus leads on construction baselines. Among the 14 tool-free configurations, Astra high leads both groups. Thus the group rankings identify different strengths, while Overall summarises proportional quality across all instances. We will foreground the separate rankings so that improvements over construction baselines are clearly distinguished from performance against published frontiers.

Output budgets. The tool-free output-budget stop counts are:

ConfigurationStops out of 69
GPT-6 Astra medium / high0 / 0
GPT-5.6 Luna medium / high1 / 1
Claude Fable 5.1 medium / high9 / 32
Claude Opus 5.5 medium / high1 / 6
DeepSeek V4.1 Flash low / high4 / 5
Qwen3.5 4B / 9B / 27B high13 / 8 / 7
Qwen3.8 27B high14

In 30 of Fable high's 32 budget stops, the completed response reports 128,000 thinking tokens and no non-thinking output: the allowance is consumed in reasoning, before a final construction is produced.

Fable medium/high scores are 71.90/43.73 on all 69 instances, versus 73.68/81.28 on the same 36 instances completed without truncation by both. This reversal shows that the effort comparison reflects completion under the specified budget as well as construction quality. The subset comparison is conditional on completion; the primary scores retain all 69 instances and budget-stop zeroes.

Search sensitivity. The 10-to-600-second experiment quantifies the budget sensitivity of the tested reference algorithms. A comparison across algorithms would assess how broadly this behaviour extends. We will use “budget sensitivity of the evaluated reference searches” in place of “search-resistant.”

Reviewer_Wd7DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents The Endless Exam, a benchmark of parameterised families of mathematical construction problem, comprising of 69 parameter instances from the 14 problems. The tasks require specific mathematical objects as answers, where some aspect of those objects is to be optimised. If the object passes programmatic validation, the model scores the ratio of the target value against a pre-defined baseline. The benchmark evaluates 9 models, including the state-of-the-art, both with 1-shot prompting and in agentic harnesses, showing GPT-6 Astra to compete with Claude Opus 5.5, depending on the setting. The authors provide further analysis, into the computational budget sensitivity, how much of the theoretical gap is resolved, as well as analysing for how problem size affects the measured quality.

Critical strengths
  1. The idea of benchmarks going beyond human limitations is key and important. As LLMs start advancing open problems, they should no longer be limited by existing discoveries.

  2. The evaluation covers a wide range of models, including state-of-the-art ones, and differentiates between one-shot evaluations and agentic harnesses.

Critical weaknesses
  1. As a metric, the overall score as the mean of the relative quality scores does not hold up. It is first of all heavily reliant on the available baselines. If a baseline is relatively trivial, this problem will hold significantly more weight than others, as it takes significantly less reasoning to be matched or improved upon, creating artificially high scores. It is also heavily prone to outliers, as significant improvements on one specific instance with high h(p)/ah(p)/a may cause a sharp increase in the mean. For example, if model A is consistently better than model B, but model B is significantly better than model A on one specific outlier, results would show model B being ahead. Since 39/69 instances do not use established baselines, this problem becomes a significant concern.

  2. Novelty remains limited, while the problem suit is small. MathConstruct already performs essentially the same methodology, but on simpler problems. The authors present the advantage with the fact that the resulting score is uncapped, the issue with which I have described in W1, as well as the research-level nature of the problems. While the latter is truly valuable, the suit only contains 14 problems, while it remains unclear how these specific problems were picked, whether there are more available candidates, and why only 14 were selected. Further, it seems that a different number of instances were created per problem, again biasing the mean towards those problems.

  3. As the benchmark becomes public, and advances in the field get published, evaluating with unlimited web access may give the models access to existing scores or improvements, disadvantaging previous runs. This is a problem highlighted in H.1, and although the authors did not find Opus 5.5 to have accessed previous public scores, it is a sandboxing issue that is never resolved.

  4. The paper does not seem to fully follow the ICLR format, as evidenced by the different header fonts in the title, sections and subsections. I am not sure whether there are any further style breaches, but I believe this warrants further elaboration.

Questions for authors
  1. In MathConstruct, the authors specifically highlighted having to select the instances, so that the required reference solution was not overwhelmingly long. For example, if constructions require thousands of tokens, it not only advantages more token-efficient models artificially, but also makes the task harder simply due to requiring long-context consistency where 1 small mistake can fully invalidate the response. Can the authors perform an audit on how this may have impacted their results, and whether programmatic outputs (in the form of generative functions if templates exist) may mitigate this, if an issue?

  2. What are the results only on the instances where non-trivial human baselines are available?

  3. Please address the concerns from the Weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wd7D

Thank you for the detailed questions. We reanalysed the original responses to examine scoring, reference strength and answer length.

Scoring and published frontiers. On the 30 published-frontier instances, tool-assisted mean relative quality is 0.9984 for Astra, 0.9851 for Opus 5.5 and 0.9609 for Luna. The 14 tool-free configurations range from 0.067 to 0.762, with Astra high first. To test dependence on baseline strength and unequal family counts, we computed:

AnalysisAstra high + toolsOpus 5.5 high + tools
Original overall, 69 instances143.16180.06
Equal weight per family134.95171.63
Log-compressed quality132.14140.33
11 stronger references132.39167.69

The log summary is 100 × (exp(mean(log(1+r))) − 1), where r is relative quality. The 11 stronger references (five LABS, five Heilbronn, one Shannon) pass both verifiers and replace the same denominators for every configuration; all remaining references stay unchanged. All 17 ranks are preserved under this partial reference replacement.

These checks directly test family weighting, outlier influence and reference strength. Opus's overall advantage is concentrated in Heilbronn and length-64 trifference. Overall summarises construction quality relative to each instance's reference. The difficulty of achieving an improvement depends on the task and reference.

Contribution and selection. MathConstruct checks whether a construction meets a specified goal; Endless Exam measures how good valid constructions are on research optimisation problems. Astra produces 69/69 valid answers in both tracks, yet its score rises from 91.90 to 143.16 with tools. This measures progress after correctness has saturated.

The selection criteria are an optimisable objective, deterministic verification, variable parameters and feasible answer representations. The initial release covers fourteen families across sets, geometry, graphs, codes and designs; further families can be added under these criteria. The 69 distinct instances cover different parameters and variants, including square and triangular Heilbronn domains. Their unequal counts motivate the equal-family analysis above.

Answer length and generative formats. Across the 14 tool-free configurations, median parsed construction size is 257–1,115 JSON bytes; 31 of 859 parsed answers exceed 10,000 bytes. Product and orbit descriptions and algebraic certificates already reduce enumeration costs, including for submitted Shannon codes with over a trillion codewords. The accepted generators use these specified formats. The audit quantifies answer length, budget exhaustion and validity. A controlled comparison of representations would isolate the contribution of long-output consistency.

Web access. The evaluations used live web access on different dates, so the available resources could differ. Benchmark materials were online during Opus's 22–23 September evaluation, after Astra and Luna's 18 September runs. The Opus trace audit found no evidence of direct access to this paper or repository. These results measure code-and-web systems on the recorded dates, including their ability to retrieve and adapt available constructions.

Formatting. Our ICLR style and bibliography files match the official archive, whose title and section definitions use small capitals. We will check the rendered fonts against the stock template.

The revision will include these analyses and clarify family selection, instance allocation and the dated web-access conditions.

We hope these tests and clarifications address your concerns about scoring and the contribution beyond correctness-based evaluation.

I thank the authors for their response. First, I would say I'd consider the following points resolved:

  • Novelty: The authors have convinced me that judging grade quality outside of binary correctness is sufficiently novel and is a good approach.
  • Formatting: I believe the initial version I received had a different style, but regardless it did not change my feeling of the paper and I would consider this issue resolved now, thanks!
  • (Partially) Scoring: I thank the authors for their response to Q2, and for trying out different scoring changes. I point out some remaining clarifications I need below.

I have some remaining questions/concerns I would like additional elaboration/clarifications on:

  • Non-trivial baseline comparisons: As in the answer for Q2, the Astra+tools and Opus+tools ranks are actually inverse, even if the full results suggest a large gap in favour of Opus. Could you provide a more in-depth analysis of this discrepancy and partially whether it is accounted by the outlier/weighting problems I discuss in the original review?
  • Scoring modifications: The equal weight and log-compressed quality measures indeed help address my concerns. However (1) they seem to have been considered separately, and (2) the logarithm is only a mild compression depending on the scales of the metrics. Can the authors produce a result that is a combination of these fixes, and what are they considering to be the main metric they present in the next revision of the paper?
  • Problem sourcing: While the criteria for what a task family must satisfy are clear, what remains unclear is (1) where these problems were sourced from and how they were identified, not how they were filtered, (2) whether more classes can be easily created, from ongoing research or otherwise, (3) how many candidates were considered, and what was the limitation for only picking 14, (4) how the number of instances per family was decided (given its broad range), and (5) how much work (manual and automated) would require to extend to 30 families let's say (in terms of hours or human labour approximately, given the authors' experience).
  • Web access: The authors' response seem to only reaffirm that the current results are valid, which I would agree with and do not argue against. However, a benchmark should also be maintainable over time in the future, even if not by the authors. How would you provide an environment that is able to run comparably in terms of permissions, and model capabilities, without providing full internet access that can search the public results and answers from other models? I ask for a clear explanation of how one would achieve this, with concrete mechanisms with technical details, and not for an actual implementation. I believe this an important point in benchmark development, and should be clarified, at least for future support.
  • Answer lengths: Can you provide these numbers in tokens (what the models actually care about),? Furthermore, in Table 19, you mention the largest answer is 77k tokens long (with a max model budget of 128k output tokens). Did any model submit an accepted solution to this problem, and if so which, and how long were their outputs? I am slightly concerned this may be a limiting factor that can either artificially penalise models with smaller output sizes (though all have the same 128k now, future models might not), or models that first generate the example in the thinking trace, scoring 0, compared to simpler models producing an ineffective but non-zero-scoring smaller example (it's good to check for this across answers anyways). If time permits, a per-instance table of answer lengths (references and models) would be appreciated.
AuthorsOpenReview ↗

Follow-up on scoring, family selection, web access, and answer length

Thank you for the follow-up. We computed the combined score you suggested and counted tokens in the saved constructions.

Scoring. With tools, Astra leads Opus on the 30 published-frontier instances (mean relative quality 0.9984 vs. 0.9851), while Opus leads on the 39 construction-baseline instances (2.4279 vs. 1.7648). On the overall score scale, these groups contribute 0.58 points in Astra’s favour and 37.48 in Opus’s favour. Of Opus’s net 36.90-point lead, 26.09 points come from length-64 trifference and 11.57 from the ten Heilbronn instances. On the remaining 58 instances, Astra is slightly ahead (117.93 vs. 117.03).

For the combined check, we averaged log(1 + relative quality) within each family, weighted the 14 family means equally, then transformed the result back to the score scale. Opus scores 129.38, Astra 123.95, and Luna 101.47. Opus remains first, but its lead over Astra narrows to 5.43 points. Overall remains the primary score for bench-v1.0. It measures average construction quality relative to each instance’s reference. On the narrower question of which system comes closest to published results, Astra leads.

Family selection. We identified candidates from published extremal constructions and their generalisations. For example, work by Polak and Schrijver on odd cycles and by Bishnoi et al. on trifference codes gave us concrete constructions to build on. We consolidated 15 initial task labels into 12 core families: we grouped the two spherical-code variants, grouped the two Heilbronn domains, and kept Kakeya as a control. In an extension round we considered seven directions and piloted three at four settings each: Shannon codes, trifference codes, and Zarankiewicz matrices. The first two joined the 12 core families; Zarankiewicz matrices remain a candidate. Each distinct parameter setting counts once. For example, Heilbronn contributes five square and five triangular instances, while trifference contributes four code lengths.

Reaching 30 families would require 16 more additions. We did not log person-hours. In the three-family pilot, the main hands-on work was sourcing reference constructions, designing answer formats, and implementing independent checkers; instance generation and verification were automated.

Web access. For future controlled comparisons, we propose keeping the 69 released instances public and evaluating models on the same held-out parameter settings before releasing any submissions. Each model would receive the same code tools and resource limits. The sandbox would have no direct internet access. All models would search the same dated literature snapshot, which other evaluators could reuse. A model could retrieve a published method and adapt it to a new instance, but later benchmark submissions would not enter its search results. The existing live-web scores describe performance on their recorded evaluation dates.

Answer lengths. We counted each serialised construction with both tokenizers in Table 19 and took the larger count. Across 859 parseable tool-free answers, the median per configuration ranged from 136 to 645 tokens; 27 answers exceeded 10,000. These counts exclude reasoning and surrounding prose.

The 77,528-token reference is a 2,564-point kissing configuration in dimension 15. Thirteen of 14 tool-free configurations returned valid constructions. Luna high submitted 420 points in 47 construction tokens; Astra high submitted 1,710 in 3,617; Fable medium and Opus high each submitted 2,340 in 70,745. Their total outputs including reasoning were 5,660, 6,758, 96,528 and 95,755 tokens, respectively. All three tool-assisted configurations matched the reference via saved files. Compact answers could therefore score on this instance, while the two highest-scoring tool-free submissions each used more than 95,000 total output tokens.

In 30 of Fable high’s 32 budget stops, the completed response reported 128,000 thinking tokens and no final-answer tokens. The raw reasoning was unavailable, so we could not check whether it contained a valid construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.