- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work introduces a benchmark for searching for lattice points configurations with no integral Radon partitions. Mainly two targets are considered 11 points in 3D and 21 points in 4D. The paper compares multiple plausible approaches on this problem. None of the methods finds a valid configuration at either target.
Critical strengths
- A concrete and relevant task.
- The authors distinguish unsuccessful searches from nonexistence proofs and disclose differences in search domains and computational budgets. Overall the results seem to be reported carefully.
- The considered approaches cover a broad spectrum: classical search, RL-guided tree search and LLM based program evolution etc.
Critical weaknesses
-
The evaluation provides very limited insights. All methods fail on the primary criterion, and it is unclear to me if a solution exists at all. Hence, it is difficult to interpret the results and distinguish whether the failure is due to an algorithm’s inductive bias, configurations, or just the nonexistence of a solution.
-
Furthermore, the task coverage is very narrow. The authors only consider open targets that are above known feasible cardinalities. I think the inclusion of known solvable tasks could improve the work and maybe help establish some meaningful differences between the approaches.
-
The presentation is very hard to follow in places.
Questions for authors
- Which specific capabilities of AI models is Randobench intended to evaluate?
- Do you have results for easier, solvable targets? How did you tune the methods compared on the presented targets?
Response to Reviewer QCxx
Thank you for recognizing the concrete task, the careful reporting, and the breadth of methods, and for your questions on the scope of the evaluation. We respond below with:
[Q1] Which specific capabilities of AI models is RadonBench intended to evaluate?
[A1] Essentially we present a case study of a very difficult open problem in combinatorial geometry, on which none of the classical or ML/AI search methods we tested found a solution. The intended capability is geometric discovery beyond known constructions: whether a method can build a valid configuration larger than the best-known one, as judged by an exact verifier rather than an approximate score.
[Q2] Do you have results for easier, solvable targets?
[A2] No, not as separately declared targets; every reported run targets the open sizes. The only related evidence is a by-product of those runs. The two constructors start from no points (PatternBoost) or only the origin (AlphaZero-Radon), and both reach the best-known sizes, 10 points in Z3 and 20 in Z4 (Table 3, kmax). The three fixed-target methods were not run on solvable targets. We agree that results on solvable targets would be useful as positive controls, and we plan to add them.
[Q3] How did you tune the methods?
[A3] We did not run a hyperparameter search for any method. All settings are fixed, listed in Tables 5–8, and shared across the 3D and 4D instances, so each reported result reflects a single untuned configuration per method.
[Q4] All methods fail and it is unclear whether a solution exists, so one cannot tell whether a failure comes from inductive bias, configuration, or nonexistence.
[A4] We agree that this is the main limitation of evaluating an open problem: when no method succeeds, a search failure cannot be separated from nonexistence of a mathematical example. The constructors' rediscovery of the best-known sizes shows that their pipelines work, but it does not resolve this ambiguity. The paper makes no nonexistence claim (Sec. 1, l. 106–107).
[Q5] The presentation is hard to follow in places.
[A5] We will add a plain-language overview of the problem and the evaluation before the technical sections.
Thank you again for your constructive questions!