acceptodds
Withdrawn from ICLR 2027

Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly participating in complex real-world tasks in the form of algorithm-design agents, designing and refining algorithms. Many successful algorithm-design agents adopt pure in-context evolutionary frameworks, but they may quickly plateau in domains that require specialized knowledge. Parametric adaptation offers a way to internalize specialized knowledge, but conventional training requires abundant domain-specific corpora while high-quality algorithms are scarce in complex algorithm-design scenarios. In this paper, we propose sample-efficient parametric self-evolution where agents can explore and learn from self-generated algorithms. First, we characterize in-context evolutionary stagnation and analytically propose the Improvement Chain proposition, showing how learning successive self-generated algorithms can locally increase the likelihood of neighboring algorithms. Motivated by this local-transfer perspective, we further propose Population-Curated Policy Optimization (PCPO) to utilize a global population and a hybrid policy update scheme for retaining and reusing high-quality, diverse self-generated algorithms, shifting the policy towards stronger algorithms. In the task of learning rate schedule design for global placement in electronic design automation, trained only on 4 chip cases, PCPO outperforms the state-of-the-art in-context evolutionary methods (e.g., OpenEvolve and ShinkaEvolve) on average across 16 chip cases. With an 8B-size base model, PCPO achieves competitive performance compared to frontier closed-source models such as GPT-5.5. PCPO also reduces inference-time token cost by internalizing grounded domain knowledge and prompt distillation. Moreover, PCPO achieves significant speedups on four GPU kernel designs, with an average of 8.27 speedup against the PyTorch Eager baseline. Our code is available at https://anonymous.4open.science/r/PCPO-222D/.

Primary area:
reinforcement learning
Keywords:
Self-evolving Agent, Evolutionary Learning, Reinforcement Learning, Automated Algorithm Design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7BcrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Population-Curated Policy Optimization (PCPO) for self-evolving algorithm-design agents. It is motivated by "in-context evolutionary stagnation", where OpenEvolve and ShinkaEvolve with Qwen3-8B underperform direct sampling, and by a conditional Improvement Chain proposition: training on one program can raise the likelihood of its neighbours. PCPO keeps a deduplicated per-instance population of verified programs ranked by reward and diversity, mixing on-policy GRPO with periodic off-policy updates on top-k elites. The authors demonstrate on Qwen3-8B that it improves HPWL by 1.54% on average over human-set parameters across 16 ISPD 2005/ICCAD 2015 chips, comparable to GPT-5.5. On 4 KernelBench Level-1 tasks it reports an 8.27× average speedup.

Critical strengths

The paper addresses an important question of how to escape in-context evolution stagnation.

Critical weaknesses

The biggest concern is that it is not clear anything is learned beyond a good schedule. The "algorithm" is one short adjust_learning_rate function over eight scalars, trained for 16 steps on 4 chips.

The stagnation diagnosis uses a weak setting: one small model as the only mutation operator and 16 to 64 rollouts, whereas AlphaEvolve and ShinkaEvolve were shown with frontier-model ensembles, and neither OpenEvolve nor ShinkaEvolve is run with GPT-5.5.

The experimental gains are difficulty to differentiate from noise. On the 12 held-out chips PCPO gains 1.33% over the default against 1.56% for GPT-5.5 best@16, a gap the paper concedes, and 1.54% against 1.64% over all 16 (Table 2).

Proposition 1 is a triangle inequality on log-probabilities and, as Section 5 concedes, says nothing about fitness.

ShinkaEvolve finds no valid placement within 16 rollouts on 5 of 16 chips (Table 1), and Codex-GPT-5.5 is 90.18% worse than the default on bigblue1 (Table 2). To me these look like harness failures, yet the paper reads them as consistent with its hypothesis. Table 6 also omits plain sampling at 64 rollouts, although Fig. 5c shows a best-of-64 Qwen3-8B sample on bigblue1 at 87.26, against 87.23 for PCPO.

Questions for authors

While I understand the authors may be resource constrained, but current experiment setup cannot back up the authors' claims empirically. Evolutionary search has very high variance, at the current experiment budget, it's impossible to differentiate any methodology gains from noise.

Reviewer_zyiLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-evolving algorithm-design agents that improve by exploring and then training on their own generated algorithms. The authors first characterize in-context evolutionary stagnation: when a task requires specialized domain knowledge, repeatedly conditioning a frozen LLM on its own flawed outputs can reinforce unproductive generation patterns. They then introduce the Improvement Chain as a motivating abstraction, with a proposition showing that, under local log-probability closeness assumptions, training on one algorithm can raise the likelihood of a neighboring one. Building on this, they propose Population-Curated Policy Optimization (PCPO). PCPO maintains a curated global population of high-quality, diverse self-generated algorithms and combines on-policy GRPO updates with periodic regularized off-policy updates on population elites, shifting the policy toward stronger algorithms. Experiments with Qwen3-8B cover two algorithm-design tasks: learning-rate schedule design for global placement in EDA (trained on 4 chips, evaluated on 16) and GPU kernel design on four KernelBench tasks.

Critical strengths
  • Insightful diagnosis of in-context stagnation. The analysis of why in-context evolution plateaus in knowledge-intensive domains is interesting. The traces in App. B.3.1 (inverted causality, sign errors, phantom API variables) make the self-reinforcing failure mode concrete.

  • Well-motivated remedy. PCPO addresses this stagnation by moving improvement into the weights: curated population elites are reused through off-policy updates. It achieves the best average rank over in-context evolutionary and RL baselines on both tasks.

Critical weaknesses
  • Weak theoretical support. Proposition 1 follows almost directly from its assumptions: a three-term triangle inequality under local log-probability closeness. The empirical check in App. B.3.2 covers one chip, one checkpoint, and four hand-made neighbors, so the Improvement Chain reads more as intuition than as support for PCPO’s design.
  • Limited, non-standard evaluation. The method is positioned against AlphaEvolve, ThetaEvolve, and TTT-Discover, but it is not tested on any of their shared benchmarks, such as circle packing, autocorrelation inequalities, or AtCoder/ALE-Bench. The kernel experiments use only four hand-picked KernelBench tasks. Results on at least one standard task would show whether PCPO’s gains, and the stagnation it targets, extend beyond chip placement.
Questions for authors

Does PCPO also improve results on standard tasks from this line of work, such as circle packing or the autocorrelation inequalities used by AlphaEvolve and ThetaEvolve? Does in-context stagnation occur on these tasks at all? Results there would help clarify whether PCPO’s gains extend beyond knowledge-intensive domains like chip placement.

Reviewer_hMBMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Population-Curated Policy Optimization, which combines on-policy GRPO with off-policy updates using high-quality, diverse programs from a persistent population. Experiments on chip global placement show that PCPO improves over several in-context evolutionary baselines and standard GRPO, with some generalization to unseen chip instances. Additional experiments on four KernelBench Level-1 tasks are provided to demonstrate applicability to kernel optimization.

Critical strengths
  • This paper is well-constructed.
  • PCPO achieves great results on global placement, including generalization to unseen chip instances.
  • The method reduces inference-time token consumption by internalizing domain-specific knowledge.
Critical weaknesses
  • The proposed off-policy objective, which assigns positive advantages to elite programs, closely resembles reward-weighted behavioral cloning. A comparison with weighted SFT using the same selected programs is needed. Additionally, the individual contribution of diversity-based replay selection remains unclear.

  • PCPO disables CoTs in its main experiments and directly optimizes generated solutions. This is very wired in RLVR settings. The paper lacks ablations comparing reasoning-enabled training or training-free methods.

  • The authors include TTT-Discover in their related work, but it is missing in their experiments. A comparison with stronger baselines like TTT-Discover is needed to establish the distinct benefits of PCPO.

  • The experiments use only four relatively simple KernelBench Level-1 tasks. Several improvements are marginal (e.g., 0.17% over GPT-5.5 on MinGPTNewGelu), without reported measurement variance. Evaluations on Level-2/3 tasks with more reliable benchmarking would strengthen the claims.

  • The Improvement Chain proposition relies on strong assumptions and does not guarantee fitness improvement. Moreover, PCPO does not explicitly construct or follow such chains, leaving the connection between the theory and the proposed algorithm unclear.

Questions for authors

See above claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.