acceptodds
Under review as a conference paper at ICLR 2027

Psi-Evolve: Autonomously Illuminating the Space of Problems

Abstract

Using modern LLMs and large amounts of resources, AI systems have become capable of solving increasingly complex existing scientific problems. However, to achieve truly autonomous scientific discovery, AI systems must also be able to propose diverse problems that go beyond readily available human knowledge. This is inherently challenging for LLMs, which have been trained on human sources and verifiable objectives, and may lack the experience and intuition to find interesting new problems with potential downstream utility. In this work, we introduce Psi-Evolve (-Evolve), new framework leveraging evolutionary computation to bridge this gap. -Evolve combines carefully designed descriptors, mutations, and objectives to find diverse problems whose solutions are inherently generalizable. Empirically, we show that -Evolve autonomously discovers archives of new problems that span well beyond the space covered by readily available human archives and whose solutions can zero-shot transfer across a given scientific domain. Furthermore, we also show that -Evolve's archives can be used in conjunction with prior AI systems, accelerating and cutting the cost of LLM-driven solution discovery itself. Together with this submission, we open-source our full implementation and the data used in its development, collected from hundreds of human problems, which we hope will support future research toward expanding the scope of autonomous scientific inquiry and discovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
problem generation, quality diversity, program evolution, open-ended, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UtAuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Ψ-Evolve, a framework for generating mathematical optimization problems using an expanding vocabulary of semantic tags and quality-diversity search. Problem quality is measured through the cross-problem transferability of independently developed solvers. Experiments across three mathematical domains evaluate problem diversity, transfer to literature-derived problems, and expert judgments.

Critical strengths
  1. Combining online semantic expansion with transfer-based problem selection provides an interesting approach to generating diverse problems with reusable solutions.

  2. The evaluation includes three domains, multiple seeds, extensive ablations, and a blinded expert study, providing useful evidence beyond automatic diversity metrics.

Critical weaknesses
  1. The CVT-MAP-Elites baseline uses global top-25 fitness selection rather than niche-wise elitism (Table 17 and released code). Some POET runs also suffered from a documented concurrency defect (Table 3), weakening the comparative evidence.

  2. Practical cost savings are not demonstrated end to end. The main transfer scores use best-of-100 solver selection, while the reported saved optimization steps do not account for archive construction and target-side selection costs.

Questions for authors
  1. Do the conclusions persist with genuine niche-wise CVT-MAP-Elites selection and reruns of the POET experiments affected by the concurrency defect?

  2. Does Ψ-Evolve reduce total solving cost relative to from-scratch ShinkaEvolve when archive construction, its amortization across tasks, and target-side retrieval/evaluation are included?

4 comments

We thank the reviewer for recognising both the contribution of our method to autonomous problem discovery and the breadth of our experimental evaluation. Below, we present additional experiments and clarifications to address the questions raised and provide a fuller understanding of our method and results.

Weakness 1 + question1:

The CVT-MAP-Elites baseline uses global top-25 fitness selection rather than niche-wise elitism (Table 17 and released code). Some POET runs also suffered from a documented concurrency defect

Do the conclusions persist with genuine niche-wise CVT-MAP-Elites selection?

Following the reviewer’s suggestion, we evaluated CVT-MAP-Elites with classical niche elitism, defining niches as cells in the induced-solver embedding space. We ran this experiment across three seeds in the geometric packing domain.

We added these new results to a new Appendix D.15 and made them also available at this link for easier reference.

In the previous plot we observe that the variant using niche elitism achieved a score of 52.352.3, compared with 56.456.4 for our baseline variant, which selects the top 2525 problems ranked by fitness. We hypothesise that this decrease is due to the smaller pool of available parents under niche elitism: approximately 1313-1616 at the end of a run, compared with 2525 for the baseline. We believe these new results will further reinforce the empirical evidence toward our new method, and confirm that Psi-Evolve’s superiority does persist compared to niche-wise CVT-MAP-Elites selection.

question1(second part)

Do the conclusions persist with genuine niche-wise [...] reruns of the POET experiments affected by the concurrency defect?

As the reviewer correctly noted, our POET implementation suffered from documented concurrency defects in two domains: geometric packing and spectral design. We reran the experiments in these domains using the fixes previously applied to the geometry of numbers domain.

We show this a plot the change between the published and updated version of POET in geometric packing and spectral design.

We replaced the POET results in the paper with the corrected ones and updated the corresponding plots:

In geometric packing, POET’s normalised score decreased from 57.757.7 to 47.047.0, while in spectral design it increased from 62.162.1 to 66.666.6. We believe these updates confirm our overall conclusions and further reinforce our results, strengthening Ψ-Evolve’s advantage over POET in geometric packing.

Weakness 2 + question2:

Practical cost savings are not demonstrated end to end.

Does Ψ-Evolve reduce total solving cost relative to from-scratch ShinkaEvolve

The reviewer also suggested a more complete comparison of the cost of Ψ-Evolve, including archive construction, with the cost of solving problems from scratch using ShinkaEvolve. We therefore conducted a detailed comparison on problems from the geometric packing domain. As reported in Table 15, the average construction cost for the 100100-problem archive is approximately 423,000423,000 tokens per archive entry. Archive construction is the dominant cost. Subsequent reuse involves retrieving and evaluating existing solvers, whose executions can be parallelised. Furthermore, Appendix D.8 shows that tag-based retrieval allows us to evaluate only half the archive while retaining 95.695.6% of the full-archive oracle score.

Among the crawled problems used to compare zero-shot performance with optimisation from scratch, 3636% are relatively easy: ShinkaEvolve requires only zero or one optimisation steps to match the zero-shot archive score. Among the remaining, more challenging problems, an average of 1717% require more than 423,000423,000 tokens of optimization from scratch to reach the archive score, with several requiring more than one million tokens. Moreover, on 77% of the evaluated problems, optimization from scratch does not exceed the zero-shot archive score even when we double the generation budget from 1515 to 3030. These new results, which we added to the latest revision, suggest that, when the goal is to match the archive's performance on every target, including the most difficult ones, optimization from scratch can become costly. On the harder problems, reaching the same performance without Psi-Evolve’s archive can require more tokens than the construction cost of the archive itself, and on some problems it does not reach the archive's performance at all, even with a doubled budget. The archive's cost, in contrast, is paid once and amortized across all subsequent targets. . Taken together, we believe these results highlight the potential of Ψ-Evolve to reduce cumulative solving costs through repeated archive reuse. This makes archive reuse particularly promising for workloads that require strong performance across both easy and challenging targets, with the overall savings depending on the frequency of reuse and the costs of retrieval and execution. We believe these results further illustrate the cost-saving potential of our method on challenging problems and reinforce the paper's overall analysis.

We conduct these discussion in Appendix D.9.

We appreciate the reviewer’s constructive comments and welcome the opportunity to continue the discussion in case they have any further questions or comments.

Summary of changes to the paper. All new or changed text is shown in blue in the revised PDF.

(W1/Q1, CVT-MAP-Elites) We added Appendix D.15, "CVT-MAP-Elites". It reports the niche-elitism variant (one elite per cell of the induced-solver embedding space) on geometric packing over three seeds and compares it in a new figure with our global top-25 baseline.

(W1/Q1, POET) We replaced the POET results for geometric packing and spectral design with the rerun results everywhere they appear:

  • the main-text figures (zero-shot and few-shot results, vocabulary growth, embedding projections, three-domain average) and the related text;
  • the POET run statistics (Table 3, Appendix C);
  • the tables and the distance-vs-transfer figure in Appendix D.7;
  • the validity table (Appendix D.13);
  • the cost and scaling tables (Tables 15 and 16, Appendix G).

(W2/Q2) We added Appendix D.9, "Practical archive reuse and end-to-end cost", which is referenced in Section 3. It compares the cost of building the archive (Table 15) and of practical solver selection (tag-based retrieval, Appendix D.8) with the effort from-scratch ShinkaEvolve needs to reach the archive's zero-shot score.

AuthorsOpenReview ↗

Kind Request for Your Feedback on Our Rebuttal

Thank you again for the time and care you have devoted to reviewing our work. With approximately one day remaining in the author/reviewer discussion period, we would like to follow up on our rebuttal and revised manuscript. We would appreciate knowing whether our responses have addressed your concerns or whether any points require further clarification. We remain available to answer any additional questions and will respond promptly. Thank you for your time and consideration.

Reviewer_bnrgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Psi-Evolve, a quality-diversity framework for autonomous problem-space exploration. The method combines an evolving semantic descriptor space with a transfer-based fitness function that evaluates generated problems based on the generalizability of their induced solvers. The authors evaluate the approach across three mathematical optimization domains, demonstrating improved problem diversity and zero-shot solver transfer compared with existing baselines. The validation includes component ablations, robustness experiments, and blinded expert evaluations.

Critical strengths
  • Interesting problem formulation and meaningful methodological contribution. The paper introduces a useful perspective on autonomous problem discovery by evaluating problems through the transferability of their induced solvers. Combining this objective with adaptive semantic exploration offers a promising direction beyond conventional difficulty-based or novelty-based problem generation.

  • Comprehensive empirical validation. The experiments cover three mathematical optimization domains and demonstrate improvements in problem diversity and solver transfer. The component ablations, robustness evaluations, and blinded expert study provide supporting evidence for the effectiveness and practical relevance of the proposed approach.

Critical weaknesses
  • Insufficient isolation of the core methodological contributions. While the ablations support the importance of semantic exploration, the specific benefit of transfer-based fitness over alternative problem-quality objectives remains unclear. Moreover, the experiments with fixed semantic vocabularies suggest that much of the transfer performance can be recovered without online descriptor expansion. More controlled comparisons would help establish the necessity of these design choices.

  • Reliance on oracle solver selection. The main zero-shot evaluation selects the best-performing solver from the entire archive, which may overestimate the performance achievable in practical settings. Although the authors provide retrieval-based evaluations, the performance decreases substantially under smaller retrieval budgets. Consequently, the claimed benefits for efficient solver reuse and accelerated solution discovery are not yet fully established.

Questions for authors
  1. Could the authors compare transfer-based fitness with alternative quality objectives, such as problem difficulty, self-performance, or novelty, while keeping the semantic exploration mechanism and computational budget unchanged? This would help clarify whether the proposed fitness function is essential to the observed improvements.

  2. Can the authors provide a more complete evaluation of practical solver reuse, including retrieval costs, solver execution costs, and comparisons with optimization from scratch under matched computational budgets? This would help establish whether the archive provides meaningful end-to-end acceleration beyond oracle selection.

5 comments

We thank the reviewer for their appreciation of our methodological contribution to the domain of autonomous problem discovery and the breadth of empirical validations. In the following, we would like to address all questions raised by the reviewer and summarize several additions we made to the latest revision of our work:

Weakness 1 + question1:

Insufficient isolation of the core methodological contributions. [...]the specific benefit of transfer-based fitness over alternative problem-quality objectives remains unclear.

Could the authors compare transfer-based fitness with alternative quality objectives?

The reviewer suggested a comparison with other Ψ-Evolve variants with different fitness functions in order to better understand the importance of the current choice of fitness function.

Following the reviewer’s request, we tested Ψ-Evolve with three different fitness functions:

  • Difficulty: this fitness function is constructed using gpt-5.6-luna (medium effort) with one attempt per problem; it writes a solver from the problem statement alone. If its score s is 0, fitness is 0; otherwise fitness = true_best / s. True_best is the oracle score of the best solver among the ones generated before for the other problems. It is the closest to Ψ-Evolve and fairly consistent across seeds.
  • Self-performance: this fitness function is measured using max(0, Shinka's best − score of the starting program) / true_best, it measures problems on which Shinka can make a maximum of progress.
  • Novelty: this fitness function is constructed by averaging the embedding distance (cosine) to the 5 nearest problems generated before it. It is fixed when the problem is created.

We added these new results to a new Appendix D.6 and made them also available at this link for easier reference.

As summarized by the plot above, we recorded notably lower performance for each of the modifications, with a normalised zero-shot score going from 6666 for the classical Ψ-Evolve to 60.460.4 for the one using difficulty, 48.748.7 if we use self-performance, and 5858 for novelty. Thus, from our experiments the transferability-based fitness function appears essential to the observed improvements.

Weakness 2:

fixed semantic vocabularies suggest that much of the transfer performance can be recovered without online descriptor expansion

This experiment referred to by the reviewer is one ablation of the online descriptor expansion in which we make an LLM generate the problem descriptors first and then use them in problem generation without adding new descriptors. Figure 8, which summarises the findings, reports both the zero-shot score on human crawled held-out problems and the distance in the embedding space to those problems. As the reviewer rightly pointed out, increasing the number of fixed online descriptors raises the average (over three seeds) zero-shot score close to that of Ψ-Evolve, with 0.570.57 for 1010 descriptors to 0.710.71 for 100100 descriptors against 0.720.72 for Ψ-Evolve. However, we would like to highlight one effect that counterbalanced the impression that “the transfer performance can be recovered without online descriptor expansion”. As we increase the number of fixed online descriptors, the distance of generated problems to the human held-out ones decreases, from 0.6170.617 for 1010 descriptors to 0.5920.592 for 100100 descriptors, far from the 0.6460.646 of Ψ-Evolve. Thus, the ablated version tends to generate problems that are closer to the ones already found by humans and less creative than the ones found by Ψ-Evolve. If you define the tags beforehand based on the domain, the LLMs tend to generate tags close to what has been generated by humans and thus artificially increase the score on the held-out dataset. It is especially staggering when looking at the best seed of fixed 100100 descriptors that score 0.810.81 but is also the closest to human data, with a distance of 0.5730.573.

Weakness 3 + question2:

Reliance on oracle solver selection.

Can the authors provide a more complete evaluation of practical solver reuse?

To address this, we extended our comparison with from-scratch ShinkaEvolve on the geometric-packing domain, distinguishing the upfront cost of archive construction, practical solver selection, and the optimization effort needed to reproduce the archive’s zero-shot performance. In our paper, Table 15 reports an average of 42.342.3 million tokens to construct an archive of approximately 100100 problems and their associated solvers, corresponding to approximately 423,000423,000 tokens per archive entry. The same runs require 10.810.8 CPU hours and 34.034.0 wall-clock hours on average. This construction cost is incurred once and can be amortized over subsequent target problems; the appropriate allocation per downstream target therefore depends on how many times the archive is reused. In Appendix D.8 we show that using tag-based retrieval we retain 95.695.6% of the full-archive oracle score while evaluating only half of the solvers. Moreover, full reuse requires no target-specific solver optimisation, and evaluations can run in parallel. We conduct an additional analysis to measure the effort required for from-scratch ShinkaEvolve to reach the zero-shot archive score. On 3636% of the evaluated problems, this requires only zero or one optimization generation, leaving limited scope for savings on these easy targets. Across the remaining problems, 1717% require more than 423,000423,000 tokens to reach the archive score, with several requiring more than one million tokens. Furthermore, on 77% of the evaluated problems, from-scratch optimization does not exceed the zero-shot archive score even after increasing its budget from 1515 to 3030 generations. These results indicate that the archive is particularly useful for difficult targets, where an existing solver can provide performance that otherwise requires substantial target-specific optimization. They also appear to show that practical retrieval preserves most of the archive’s aggregate utility. Taken together, we believe these results support the practical value of archive reuse beyond exhaustive oracle selection. Tag-based retrieval preserves most of the archive’s aggregate performance, while the comparison with from-scratch optimisation shows that existing solvers can avoid substantial target-specific optimisation on challenging problems. This makes Ψ-Evolve a promising resource for repeated problem solving within a domain, particularly when the upfront construction cost is amortised across many downstream tasks.

We thank the reviewer again for their thoughtful feedback and would be happy to address any further questions or clarify any remaining points.

Summary of changes to the paper. All new or changed text is shown in blue in the revised PDF.

(W1/Q1) We added Appendix D.6, "Choice of fitness function". It defines the three alternative fitness functions (difficulty, self-performance and novelty) and compares them with the transferability-based fitness in a new figure.

(W2) We did not change the paper for this point. The fixed-vocabulary ablation and how it trades transfer against distance to the literature are discussed in Appendix D.3 (Figure 8).

(W3/Q2) We added Appendix D.9, "Practical archive reuse and end-to-end cost", which is referenced in Section 3. It compares the cost of building the archive and selecting solvers in practice with the effort from-scratch ShinkaEvolve needs to reach the archive's zero-shot score. The retrieval results it builds on are in Appendix D.8.

Reviewer_jYBHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Psi-Evolve, a framework for autonomously exploring scientific problem spaces. Experiments across three mathematical domains show improvements in problem diversity and zero-shot solver transfer. I find the research direction interesting and important, particularly the shift from solving predefined problems to autonomously discovering new ones. However, several aspects of the experimental evaluation limit the strength of the conclusions.

Critical strengths

Important research direction. Moving beyond solving human-defined problems toward autonomous problem discovery is a meaningful step for AI-driven scientific research.

Interesting methodological design. Combining adaptive semantic exploration with solver-transfer-based problem selection provides a reasonable approach to balancing diversity and potential utility.

Comprehensive experimental effort. The paper evaluates three mathematical domains and includes comparisons with established approaches, component ablations, computational cost analysis, and a blinded human expert study.

Critical weaknesses

Solver transferability does not necessarily reflect scientific value. The central assumption is that a problem is valuable if its induced solver generalizes to other problems. However, broadly transferable algorithms do not necessarily imply scientifically meaningful or novel problems.

The main evaluation relies on oracle solver selection. The reported zero-shot performance selects the best solver from the entire archive for each held-out problem, requiring exhaustive evaluation of all 100 solvers. This substantially overestimates the performance achievable when encountering a new problem without knowing which solver works best. Although the authors include retrieval experiments, selecting the top 10 solvers achieves only 41.6 compared with the oracle score of 66.0.

Limited evidence of autonomous scientific discovery. All experiments are conducted in three mathematical optimization domains with predefined validity constraints, executable evaluators, and shared solver interfaces. Novelty is primarily assessed using embedding distances from literature-derived problems. Neither embedding distance nor solver transfer demonstrates that the system discovers genuinely new scientific questions or insights. The broader claims about autonomous scientific discovery therefore remain insufficiently validated.

I am open to raising my score if the authors can satisfactorily address these concerns in their rebuttal.

Questions for authors

What evidence supports using solver transferability as a proxy for scientific value, particularly given its weak correlation with expert-rated reusable insight?

Can the authors report practical end-to-end performance using a fixed retrieval budget, rather than relying primarily on oracle selection over the entire solver archive?

Can the authors provide concrete examples of generated problems that experts identify as genuinely novel and scientifically meaningful, beyond differences in problem embeddings or transferable solver performance?

5 comments

We thank the reviewer for their interest in our method and for recognising both the importance of the research direction and the comprehensiveness of our experimental effort. We made several additions to the paper in response to their feedback and tried to address each one of their concerns, as summarized in our rebuttal below:

Weakness 1 + question1:

Solver transferability does not necessarily reflect scientific value.

What evidence supports using solver transferability as a proxy for scientific value, particularly given its weak correlation with expert-rated reusable insight?

The intuition behind this choice is that problems that give rise to highly transferable solvers are more likely to address general questions than narrow, potentially contrived ones. The mutation operators promote diversity and exploration across the problem space, while the transferability-based fitness guides selection within each niche. Specifically, within an archive cell containing similar problems, this fitness favours problems whose induced solvers have broader applicability, encouraging the selection of more general questions within that niche. Moreover, following the reviewers' feedback, we conducted several additional experiments using 33 different objective functions, clarifying the importance of the transferability-based fitness function: In particular, we tested Ψ-Evolve with three different fitness functions:

  • Difficulty: this fitness function is constructed using gpt-5.6-luna (medium effort) with one attempt per problem; it writes a solver from the problem statement alone. If its score s is 0, fitness is 0; otherwise fitness = true_best / s. True_best is the oracle score of the best solver among the ones generated before for the other problems. It is the closest to Ψ-Evolve and fairly consistent across seeds.
  • Self-performance: this fitness function is measured using max(0, Shinka's best − score of the starting program) / true_best, it measures problems on which Shinka can make a maximum of progress.
  • Novelty: this fitness function is constructed by averaging the embedding distance (cosine) to the 5 nearest problems generated before it. It is fixed when the problem is created.

We added these new results to Appendix D.6 and made them also available at this link for easier reference

As summarized by the plot above, we recorded notably lower performance for each of the modifications, with a normalised zero-shot score going from 6666 for the classical Ψ-Evolve to 60.460.4 for the one using difficulty, 48.748.7 if we use self-performance, and 5858 for novelty. Thus, the transferability-based fitness function is essential to the observed improvements.

The reviewer also highlights a result from our blinded expert evaluation. In this experiment, we measured the correlation between problem fitness and experts’ assessments of the potential for a solution to yield reusable insights. We found a Spearman correlation of 0.190.19, which is positive but weak. Several considerations qualify this result. Given the limited number of experts (66), we are cautious about drawing strong conclusions from these measurements alone. Moreover, experts had to assess the potential for reusable insight based solely on the problem statement, which is a challenging task. A separate analysis covering hundreds of problems, independent of expert ratings, found a strong Pearson correlation of 0.8930.893 between internal generality fitness and performance on held-out literature-derived problems. Taken together, these results strengthen the case for solver transferability as a useful objective for autonomous problem discovery. The strong relationship between internal fitness and held-out performance supports the metric’s ability to identify problems whose solvers remain useful beyond the generated archive, while the weakly positive expert-rating correlation provides a complementary, exploratory signal. Combined with our mechanisms for promoting diversity, this objective provides a concrete basis for exploring new problems while prioritising those with broader algorithmic relevance.

Weakness 2 + question2:

The main evaluation relies on oracle solver selection. The reported zero-shot performance selects the best solver from the >entire archive for each held-out problem, requiring exhaustive evaluation of all 100 solvers.

Can the authors report practical end-to-end performance using a fixed retrieval budget, rather than relying primarily on oracle >selection over the entire solver archive?

We conducted the experiment requested by the reviewer, measuring end-to-end performance with a fixed retrieval budget. Thus, during the problem generation, any performance measurement is conducted using the 10 closest neighbours in the embedding space. We run this additional experiment in the geometric packing domain.

We added these new results to a new Appendix D.12 and made them also available at this link for easier reference.

Using this approximated fitness, we see in the plot above that the mean normalised zero-shot score using the 1010 nearest neighbors is 36.336.3 (against 41.641.6 for the original Ψ-Evolve), above the context-free problem generation (11.911.9), CVT-MAP-Elites (27.627.6) and POET (32.932.9). Thus, even with a fixed budget end-to-end our method maintains superior performances compared to the baselines.

Weakness 3 + question3:

Limited evidence of autonomous scientific discovery.

Can the authors provide concrete examples of generated problems that experts identify as genuinely novel and scientifically >meaningful, beyond differences in problem embeddings or transferable solver performance?

Following the reviewer’s feedback, we added to the latest revision an experiment where we generated 12501250 problems using Ψ-Evolve in the geometry of numbers domain. We then conducted an automated assessment using Claude Opus 5.5, with reasoning effort set to extra high and access to web search, to classify the problems into five categories:

  • known: problems with a known solution
  • novel contrived: problems that are novel only thanks to arbitrary choice of threshold or odd mixes of constraints
  • ill-posed: problems unbounded, saturated, or the objective is constant
  • open and studied: the problems are open, and there is literature on them. Some bounds or constructions about aspects of the problem have been proven or published. The problems in this category are novel in the sense that they tackle the question in a different way. Examples are lattice packing of the cross-polytope, the 4-simplex (Minkowski's problem), and ℓp-ball packing.
  • open and new: the problems are natural, but the automatic reviewer found no work on that specific body. Any good answer is new. Examples are the complex cross-polytope, the central section of the 5-cube, Ky Fan norm balls, the ℓ1-sum of two 3-balls, and Z[φ]-lattices in R⁴.
Problem typeNumber of problems (% of problems generated)
Known502 (40%)
Novel and contrived450 (36%)
Ill posed38 (3%)
Open and studied239 (19%)
Open and new21 (2% )

We believe this automated assessment suggests that Ψ-Evolve can generate a substantial number of novel and meaningful problems, both by approaching existing open questions in new ways and by proposing new questions, and reinforces our work’s empirical analysis.

We thank the reviewer once again for their insightful comments and would welcome any further questions or discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General comment

We thank the reviewers for their time, constructive feedback and their appreciation of the importance of this research direction. We answer each reviewer's questions in the individual responses. The main changes are listed below and are shown in blue in the revised paper.

  • Choice of fitness function (Appendix D.6, Figure 10). We replaced the transfer-based generality fitness with a difficulty, self-performance, or novelty fitness; all three lower zero-shot performance.

  • Practical archive reuse and end-to-end cost (Appendix D.9, referenced from Section 3). We added a comparison showing how the cost of building the whole Psi-Evolve archive would be significantly outweighed by the additional cost required for running ShinkaEvolve to recover the same final performance on zero-shot on over 25% of individual problems.

  • Solver transferability as a proxy for problem value (Appendix D.11, referenced from Section 3). We discussed why transferability is a useful objective, using the expert-rating and held-out transfer correlations.

  • Fitness under a fixed evaluation budget (Appendix D.12, Figure 12). We compute the fitness only on the 10 nearest neighbors of each problem; this variant scores 58.8, above all baselines.

  • New problem discovery (Appendix D.14, Table 10). We added new analysis showing how Claude Opus 5.5 with web search classifies 1,250 Ψ-Evolve problems in geometry of numbers: 19% address open, studied questions in new ways and 2% are open, new problems.

  • CVT-MAP-Elites with niche elitism (Appendix D.15, Figure 13). We added a new baseline, which scores 52.3, below both our top-25 CVT-MAP-Elites baseline (56.4) and Ψ-Evolve (66.0).

  • POET baseline. We updated the POET results throughout the paper. We replaced the POET results for geometric packing and spectral design with the rerun results everywhere they appear. Thus, we update the main-text figures (zero-shot and few-shot results, vocabulary growth, embedding projections, three-domain average) and the related text and the POET related statistics in the Appendix. That does not change the previously given conclusions and sometimes strengthens them.

  • Figure 11. We corrected the caption of the distance-vs-transfer figure.

AuthorsOpenReview ↗

Kind Request for Your Feedback on Our Rebuttal

We thank all reviewers for their thoughtful comments and constructive feedback. With less than 24 hours remaining in the author–reviewer discussion period, we would greatly appreciate any further feedback on our responses to your comments. If any concerns remain or additional clarification would be helpful, please let us know, and we will respond promptly. Thank you again for your time and consideration.

Dear Reviewers,

As the discussion is clouding soon, could you please read the authors responses and respond to them for any feedback you have? Thanks!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.