- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Endless Exam, a benchmark of 69 instances from 14 families of mathematical construction problems. It uses automatic verification, compact certificates for large constructions, and quality scores relative to published frontiers or construction baselines. The evaluation covers nine models with different reasoning efforts and tool settings, and examines how construction quality changes with problem size.
Critical strengths
The model coverage is broad, including commercial and open models, different model sizes, reasoning efforts, and tool settings. This provides a useful comparison of current systems.
The verification and released materials are thorough. The code and saved results are consistent with the main reported results.
Critical weaknesses
The contribution seems incremental relative to existing construction and open-problem benchmarks. The paper mainly combines existing problem families, parameter variations, and relative scoring. The 69 instance evaluation and short size sweeps provide limited support for the much broader “endless” and “superintelligence” framing. Increasing parameters alone does not establish that the resulting tasks will continue to provide useful measurements. Particularly, I think the title of this paper doesn't match the content well since there is limited evidence and discussion on "endless" and no connection with the "toward superintellegence".
The overall score is difficult to interpret. Of the 69 instances, 39 use construction baselines that may be weaker than published results. Averaging the quality ratios also gives some instances disproportionate influence: the length-64 trifference instance accounts for about 71% of Opus’s overall lead over Astra in the tool-assisted evaluation. I am not convinced that this score reliably reflects broad improvements across the suite.
Compared to prior arts, the size of the problem set is small, inconsistent with it's "endless" framing.
Questions for authors
What evidence from the current experiments supports the claim that parameter variation will sustain useful evaluation beyond current models? How do you distinguish increased mathematical difficulty from greater output requirements or limitations of the accepted representations?
What justifies averaging quality ratios across objectives with very different scaling, such as code cardinality and geometric area? Appendix H reports stability when excluding individual families, but do you have additional analysis showing sensitivity to stronger construction baselines or less outlier-sensitive aggregation?
A suggestion beyond questions: I appreciate the authors' effort in presentation and visual design. However, as in a text rich paper, I think figure 1 and those in appendix A are not necessary given the low information density they have. Those figures, although can be great visualization in slides and videos, contains mainly texts that are already described in the main body, and the illustrations don't help a lot here. I suggest the author to create a denser and more detailed main figure as figure 1.
2 comments
Thank you for answering my question. I still found the design highly heuristic and my concerns on "overclaiming" or putting irrelevant terms (superintellegence...) in the title and abstract unsolved. Given the limited size of this "endless exam", I am not convinced to upgrade my score.
Response to Reviewer rBTE
Thank you for checking the released code and results. We completed additional scoring and representation analyses using the original responses.
Contribution and scaling evidence. Endless Exam measures construction quality after validity has saturated: Astra has 69/69 valid answers in both tracks, while Overall rises from 91.90 to 143.16 with tools. Compared with correctness-based construction evaluation, this records the quality of valid solutions on research optimisation problems. Compact certificates extend the measurement to submitted codes with over a trillion codewords.
The following Luna high series each have two valid, untruncated answers at every size:
These declines occur with short completed descriptions, without output truncation. Astra's cap-set products instead retain quality 0.970 at dimension 12. The curves distinguish methods that preserve quality from those that fall behind as size grows. The 69 instances are the released evaluation set; the families generate further tasks under the same rules. Scoring beyond reference parity and expansion to larger parameters provide the concrete basis for “Endless Exam.” The title's “toward superintelligence” reflects the intended progression: measuring better constructions on existing instances, then extending evaluation to larger instances as models advance.
Aggregation, references and concentrated gains. A ratio of 1.2 means 20% more codewords or 20% larger minimum triangle area than the instance's reference. Averaging these ratios summarises relative construction quality across instances. The difficulty of achieving those gains depends on the task and reference.
Using 11 independently verified stronger references gives tool scores of 167.69 for Opus and 132.39 for Astra and preserves all 17 configuration ranks; all remaining references stay unchanged. A less outlier-sensitive summary,
100 × (exp(mean(log(1+r))) − 1)withrdenoting relative quality, gives 140.33/132.14. These directly test the reference and aggregation choices raised in your question.Length-64 trifference contributes 26.09 of Opus's 36.90-point lead, reflecting three times as many verified codewords. Opus's largest gains occur in Heilbronn and length-64 trifference, while Astra achieves the highest mean on published-frontier instances.
Representation and presentation. At trifference length 64, the current formats have an exact ceiling of 531,441 codewords, attained by Opus and documented in Appendix D.5. Scoring rewards improvements beyond reference parity within the accepted formats. In the revision, we will put these format limits beside the main description and make Figure 1 more informative by connecting the scoring and short-answer scaling evidence to the benchmark design, reducing repetitive illustrations.
We hope these results provide a clearer basis for reassessing the empirical contribution.