- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates principle-level search collapse in feedback-driven LLM optimizers, where candidate solutions continue to vary while the search concentrates on a restricted set of design principles. It introduces a diagnostic protocol combining structural concentration, prolonged performance plateaus, and gaps to attainable reference solutions. The protocol is evaluated across molecular optimization, sentinel-node selection, and GPU-kernel optimization using multiple optimizers and LLM backbones. The paper also examines diversity injection and reference-principle interventions, finding that their benefits vary across models and tasks.
Critical strengths
1.Broad empirical investigation. The study examines multiple optimization domains, algorithms, and LLM backbones, providing evidence beyond an isolated example of search stagnation.
2.Useful diagnostic controls. The experiments include repeated runs, negative controls, and kernel-specific threshold and diagnostic-window sensitivity analyses. These help distinguish concentration near competitive solutions from concentration accompanied by substantial performance gaps.
3.Informative intervention results. The reference-principle experiments illustrate that receiving a promising direction and successfully developing a competitive implementation are distinct challenges. The paper reports both improvements and unsuccessful interventions, making the empirical picture more informative.
Critical weaknesses
The incremental contribution over existing work is insufficiently established. The distinction between candidate diversity and principle diversity closely relates to established ideas concerning behavioural diversity, premature convergence, and deceptive search, as the paper itself acknowledges in Section 2. The additional insight provided by the proposed diagnostic framework therefore needs stronger justification. In particular, the experiments do not include direct comparisons with relevant route-preserving approaches such as K-Search or a principle-level archive. Such comparisons would help establish what remains unresolved beyond the observation that increasing candidate variation does not consistently prevent stagnation.
The diagnostic measurements do not sufficiently establish the formally described failure mode. Equation (1) describes the principle distribution governing candidate generation, whereas Equation (3) measures label frequencies among historical top-performing candidates. Although the authors acknowledge these measurements as proxies, concentration after selection does not necessarily imply concentration in the generation process: an optimizer may continue exploring multiple principles while only one enters the retained set. Furthermore, low final entropy in the structural diagnostic branch does not establish that the same principle remained dominant throughout the plateau. A reference gap can also reflect insufficient optimization within an otherwise promising principle. The recent-proposal analysis for kernels partially addresses these concerns, but the connection between the diagnostic labels and principle-level locking remains insufficiently validated across domains.
The plateau criterion requires clarification when observed and planned budgets differ. The appendices define the remaining-budget fraction as ((B-b_{\mathrm{plat}})/B), where (B) is the planned budget, while plateau onset is calculated relative to the observed endpoint (T_{\mathrm{obs}}). If a run terminates early, this criterion can classify it as having a long plateau even when performance continues improving until its final observed evaluation, because unused budget contributes to the fraction. The main-text kernel discussion also refers to observed attempts, creating ambiguity about the implemented criterion. The authors should clarify whether all assessed runs exhaust their planned budgets and, otherwise, separate observed stagnation duration from unused budget and report whether this changes the classifications.
Questions for authors
See Weaknesses
7 comments
Response to KrHe [2/2]
Table 4. Does the most common molecular scaffold change during the plateau?
| Backbone | Runs with the same scaffold leading every window | Other assessed runs | Proposals using the plateau's most common scaffold |
|---|---|---|---|
| GPT-5.4 | 4/5 | Two scaffolds take turns leading in 1 run | 60.1% ± 14.4% |
| Claude Sonnet 5 | 3/5 | Two scaffolds take turns leading in 2 runs | 65.1% ± 17.4% |
| DeepSeek V4 Flash | 3/5 | Two scaffolds take turns leading in 2 runs | 79.8% ± 21.3% |
| Qwen3.5 35B | 3/4 | Two scaffolds take turns leading in 1 run | 38.4% ± 17.6% |
| GPT-4o | 2/2 | None | 81.5% ± 2.5% |
For example, GPT-5.4 keeps the same scaffold as the most common one in every window in four of five runs. In the remaining run, two different scaffolds lead at different times. The last column is the fraction of valid proposals using the single most common scaffold over the whole plateau. One Qwen run and three GPT-4o runs have plateaus too short for this comparison. Thus, most assessed runs repeatedly favour the same scaffold, rather than continually switching to new directions. The exceptions alternate between two window-leading scaffolds and provide weaker evidence.
Table 5. Do dynamic-system proposals keep using the same main nodes during the plateau?
| Task | Backbone | Node-pool overlap: neighbouring windows | Node-pool overlap: later windows vs first window |
|---|---|---|---|
| Residence Hall | GPT-5.4 | 78.5% ± 9.1% | 71.6% ± 6.2% |
| Residence Hall | Claude Sonnet 5 | 89.4% ± 6.1% | 70.4% ± 15.6% |
| Residence Hall | DeepSeek V4 Flash | 88.1% ± 6.3% | 46.5% ± 13.1% |
| Residence Hall | Qwen3.5 35B | 85.8% ± 5.4% | 86.2% ± 7.2% |
| Dolphin | GPT-5.4 | 74.1% ± 1.8% | 56.6% ± 6.8% |
| Dolphin | Claude Sonnet 5 | 73.0% ± 6.0% | 61.4% ± 4.3% |
| Dolphin | DeepSeek V4 Flash | 83.0% ± 12.9% | 74.0% ± 17.3% |
| Dolphin | Qwen3.5 35B | 82.3% ± 7.1% | 80.2% ± 5.4% |
For each window, we take the ten most frequently proposed nodes. Overlap is the number of shared nodes divided by the number of nodes in either pool: 100% means identical pools. The main node pools overlap substantially throughout the plateau: overlap between neighbouring windows averages 73.0%–89.4% across settings. Later windows also retain substantial overlap with the first window (46.5%–86.2%), with some changes accumulating over time. Overall, the proposals repeatedly draw on largely overlapping node pools rather than continually replacing them with entirely different ones.
GPU kernels: generation during the plateau. For kernels, the diagnostic set is top 10. In all 49 kernel runs diagnosed as collapsed, we looked at every valid kernel generated after the plateau began. In the median run, 96% of these newly generated kernels use the same route that dominates the final ten fastest kernels (25th percentile: 88%). When we split each plateau into consecutive windows of 20 newly generated kernels, the dominant route falls below half of a window in only 4 of the 49 runs, and the most common route changes between windows in only 5 runs. About 30% of the attempts after the plateau produce invalid kernels (they fail to compile, crash, or give wrong results); these have no route label and are not counted.
GPU kernels: is the gap due to under-optimization within a promising route? For kernels we can separate the two explanations. For each of the 49 collapsed runs, we compare the run's own best latency with the best latency that any of our 455 kernel runs achieved on the same route. In 46 of the 49 runs, even that best latency is at least 2× slower than the expert kernel (median 23×), so further optimization within the route, as far as any of our runs got, would not close the gap. The run itself is a median 2.25× slower than that best latency, so some within-route under-optimization is also present. On a logarithmic scale, the route's limit accounts for a median 78% of a run's gap to the expert, and within-route under-optimization for the remaining 22%.
Please clarify whether all runs exhaust their planned budgets, separate observed stagnation from unused budget, and report whether this changes the classifications.
Response: Almost all of our runs completed their planned budgets. A small number of runs on molecular optimization tasks were stopped early because their uniqueness dropped to 10% or lower, with no improvement in the top-10 scores for 50 consecutive generations (equivalent to 2,500 generations of candidates). Continuing these runs would incur extreme time and API costs with almost no benefit. Therefore, we assume that their top-10 scores would remain unchanged until the full budget was exhausted. This only happens in molecular optimization task; all node and kernel optimization runs completed their full budgets. We will clearly state this in the paper.
Reference: [1] Pu, et.al . "Principle-evolvable scientific discovery via uncertainty minimization. ICML 2026
Dear Reviewer,
We are nearing the end of the discussion phase. Please read the authors' responses, reply to acknowledge their feedback, and indicate whether your initial concerns have been addressed (along with any score adjustments, if warranted).
Thank you for your dedicated efforts and valuable insights!
Best, Your AC
Response to KrHe [1/2]
We thank the reviewer for recognizing the cross-domain evidence and diagnostic controls. We clarify the principle-aware baseline, add within-plateau identity tracking, and report budget coverage below.
Response:
We already include PiEvo as a principle-level archive baseline across all three domains in paper. PiEvo [1] explicitly retains design principles and their associated records, maintains beliefs over the principles, and selects and revises them during optimization. The PiEvo results examine why retaining principles may still be insufficient: a direction can remain available without receiving sustained productive development, and predictable feedback can leave an unproductive direction unrevised. This is an important distinction between having an archive of principles and effectively using it to sustain exploration.
Response:
We now examine both actual proposals and structural identities throughout the observed plateau. The proposal-versus-archive analysis covers 65 original ExLLM runs, with five seeds per setting. Early/late windows contain 1,000 saved attempts, retaining repetitions among valid candidates. The contemporaneous selected-50 uses the original mixed single-/multi-objective selection rule. Support overlap C=∑j:qj>0pj measures proposal mass within this selected structural support; distribution overlap O=∑jmin(pj,qj) also compares frequencies.
Table 1. Molecular proposal concentration, repeated here for clarity.
Table 2. Molecular proposal overlap with the contemporaneous mixed-selection selected-50.
Generation itself narrows for GPT-5.4, Sonnet, DeepSeek, and GPT-4o, and substantially overlaps the selected scaffold region. Thus, the finding is not confined to the historical selected set.
Table 3. Residence Hall proposal concentration and overlap.
GPT-5.4, Sonnet, and DeepSeek narrow their generated node pools, with substantial overlap in elite node support. Node-support overlap does not require identical node frequencies.
Does the same direction remain dominant during the plateau? We divide the proposals made after plateau onset into consecutive windows of 1,000 attempts and check what is most frequently generated in each window.