- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Ψ-Evolve, a framework for generating mathematical optimization problems using an expanding vocabulary of semantic tags and quality-diversity search. Problem quality is measured through the cross-problem transferability of independently developed solvers. Experiments across three mathematical domains evaluate problem diversity, transfer to literature-derived problems, and expert judgments.
Critical strengths
-
Combining online semantic expansion with transfer-based problem selection provides an interesting approach to generating diverse problems with reusable solutions.
-
The evaluation includes three domains, multiple seeds, extensive ablations, and a blinded expert study, providing useful evidence beyond automatic diversity metrics.
Critical weaknesses
-
The CVT-MAP-Elites baseline uses global top-25 fitness selection rather than niche-wise elitism (Table 17 and released code). Some POET runs also suffered from a documented concurrency defect (Table 3), weakening the comparative evidence.
-
Practical cost savings are not demonstrated end to end. The main transfer scores use best-of-100 solver selection, while the reported saved optimization steps do not account for archive construction and target-side selection costs.
Questions for authors
-
Do the conclusions persist with genuine niche-wise CVT-MAP-Elites selection and reruns of the POET experiments affected by the concurrency defect?
-
Does Ψ-Evolve reduce total solving cost relative to from-scratch ShinkaEvolve when archive construction, its amortization across tasks, and target-side retrieval/evaluation are included?
4 comments
Weakness 2 + question2:
Practical cost savings are not demonstrated end to end.
Does Ψ-Evolve reduce total solving cost relative to from-scratch ShinkaEvolve
The reviewer also suggested a more complete comparison of the cost of Ψ-Evolve, including archive construction, with the cost of solving problems from scratch using ShinkaEvolve. We therefore conducted a detailed comparison on problems from the geometric packing domain. As reported in Table 15, the average construction cost for the -problem archive is approximately tokens per archive entry. Archive construction is the dominant cost. Subsequent reuse involves retrieving and evaluating existing solvers, whose executions can be parallelised. Furthermore, Appendix D.8 shows that tag-based retrieval allows us to evaluate only half the archive while retaining % of the full-archive oracle score.
Among the crawled problems used to compare zero-shot performance with optimisation from scratch, % are relatively easy: ShinkaEvolve requires only zero or one optimisation steps to match the zero-shot archive score. Among the remaining, more challenging problems, an average of % require more than tokens of optimization from scratch to reach the archive score, with several requiring more than one million tokens. Moreover, on % of the evaluated problems, optimization from scratch does not exceed the zero-shot archive score even when we double the generation budget from to . These new results, which we added to the latest revision, suggest that, when the goal is to match the archive's performance on every target, including the most difficult ones, optimization from scratch can become costly. On the harder problems, reaching the same performance without Psi-Evolve’s archive can require more tokens than the construction cost of the archive itself, and on some problems it does not reach the archive's performance at all, even with a doubled budget. The archive's cost, in contrast, is paid once and amortized across all subsequent targets. . Taken together, we believe these results highlight the potential of Ψ-Evolve to reduce cumulative solving costs through repeated archive reuse. This makes archive reuse particularly promising for workloads that require strong performance across both easy and challenging targets, with the overall savings depending on the frequency of reuse and the costs of retrieval and execution. We believe these results further illustrate the cost-saving potential of our method on challenging problems and reinforce the paper's overall analysis.
We conduct these discussion in Appendix D.9.
We appreciate the reviewer’s constructive comments and welcome the opportunity to continue the discussion in case they have any further questions or comments.
Summary of changes to the paper. All new or changed text is shown in blue in the revised PDF.
(W1/Q1, CVT-MAP-Elites) We added Appendix D.15, "CVT-MAP-Elites". It reports the niche-elitism variant (one elite per cell of the induced-solver embedding space) on geometric packing over three seeds and compares it in a new figure with our global top-25 baseline.
(W1/Q1, POET) We replaced the POET results for geometric packing and spectral design with the rerun results everywhere they appear:
- the main-text figures (zero-shot and few-shot results, vocabulary growth, embedding projections, three-domain average) and the related text;
- the POET run statistics (Table 3, Appendix C);
- the tables and the distance-vs-transfer figure in Appendix D.7;
- the validity table (Appendix D.13);
- the cost and scaling tables (Tables 15 and 16, Appendix G).
(W2/Q2) We added Appendix D.9, "Practical archive reuse and end-to-end cost", which is referenced in Section 3. It compares the cost of building the archive (Table 15) and of practical solver selection (tag-based retrieval, Appendix D.8) with the effort from-scratch ShinkaEvolve needs to reach the archive's zero-shot score.
Kind Request for Your Feedback on Our Rebuttal
Thank you again for the time and care you have devoted to reviewing our work. With approximately one day remaining in the author/reviewer discussion period, we would like to follow up on our rebuttal and revised manuscript. We would appreciate knowing whether our responses have addressed your concerns or whether any points require further clarification. We remain available to answer any additional questions and will respond promptly. Thank you for your time and consideration.
We thank the reviewer for recognising both the contribution of our method to autonomous problem discovery and the breadth of our experimental evaluation. Below, we present additional experiments and clarifications to address the questions raised and provide a fuller understanding of our method and results.
Weakness 1 + question1:
Following the reviewer’s suggestion, we evaluated CVT-MAP-Elites with classical niche elitism, defining niches as cells in the induced-solver embedding space. We ran this experiment across three seeds in the geometric packing domain.
We added these new results to a new Appendix D.15 and made them also available at this link for easier reference.
In the previous plot we observe that the variant using niche elitism achieved a score of 52.3, compared with 56.4 for our baseline variant, which selects the top 25 problems ranked by fitness. We hypothesise that this decrease is due to the smaller pool of available parents under niche elitism: approximately 13-16 at the end of a run, compared with 25 for the baseline. We believe these new results will further reinforce the empirical evidence toward our new method, and confirm that Psi-Evolve’s superiority does persist compared to niche-wise CVT-MAP-Elites selection.
question1(second part)
As the reviewer correctly noted, our POET implementation suffered from documented concurrency defects in two domains: geometric packing and spectral design. We reran the experiments in these domains using the fixes previously applied to the geometry of numbers domain.
We show this a plot the change between the published and updated version of POET in geometric packing and spectral design.
We replaced the POET results in the paper with the corrected ones and updated the corresponding plots:
In geometric packing, POET’s normalised score decreased from 57.7 to 47.0, while in spectral design it increased from 62.1 to 66.6. We believe these updates confirm our overall conclusions and further reinforce our results, strengthening Ψ-Evolve’s advantage over POET in geometric packing.