acceptodds
Under review as a conference paper at ICLR 2027

Understanding Framework Advantages in LLM-Driven Program Evolution

Abstract

LLM-driven program evolution frameworks are attracting growing attention, yet how their task-specific advantages develop and vary with models and search budgets remains insufficiently understood. We systematically compare four frameworks across five tasks in 1,560 runs under a common 50M-token request-start threshold. Pairwise and within-pool nearest-neighbor distances characterize the dispersion and local structure of task-specific execution profiles, while size-matched random subsets provide a reference for high-quality candidate concentration. We combine these measures with reconstructed budget trajectories and program-level controls to examine search improvements. Framework advantages are task-specific and broadly consistent across model configurations, although some pairings reverse rankings. Selecting a framework for each configuration does not consistently outperform a task-wide choice on held-out search repeats with the same inputs. Lower response dispersion alone does not consistently identify the best-performing framework. Reconstructed trajectories show that additional budget narrows quality gaps on some tasks and increases attainment of common quality targets. These findings support framework and model selection based jointly on task fit and available budget, rather than configuration-level rankings or response dispersion alone.

Primary area:
datasets and benchmarks
Keywords:
Large language model, Algorithm design, Evolutionary search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gyVBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares four LLM-driven program evolution frameworks across five tasks and 26 model/reasoning configurations, reporting 1,560 runs under a common 50M-token request-start threshold. It examines task-dependent framework advantages, model effects, and budget-dependent progress, supplemented by execution-profile analysis and selected program-level controls.

Critical strengths
  • The comparison covers multiple frameworks, tasks, and model configurations, providing a broad view of performance under the evaluated settings.
  • The paper distinguishes observed rankings from selection gains on held-out search repeats. It also explicitly acknowledges the limits of fixed-input evaluation, reconstructed budget coordinates, and program-level attribution.
Critical weaknesses

The paper's additional insight beyond prior work remains unclear.

The main conclusions are that framework advantages depend on tasks, models, and budgets, and that response dispersion alone does not identify the best-performing framework. These findings are broadly consistent with reasonable expectations. Section 2 discusses studies addressing related factors, but the paper does not clearly identify which specific findings materially extend that understanding or provide additional guidance for framework selection. For a paper centered on comparative analysis, this incremental contribution needs clearer articulation.

Questions for authors
  1. The stated protocol implies that 3M3M for Flowshop and 9939/Q9939/Q for bin packing should be integers. Across the four frameworks, 207+183=390 of the 624 run-level records (approximately 62.5%) yield values more than 0.1 away from the nearest integer. Could the authors explain these discrepancies, including any additional aggregation, precision handling, or evaluator differences?

  2. Could the authors clarify which results in Appendix D.2, Table 16, and Figure 14 come from actual candidate evaluations and which come from simulation? The supplementary provenance identifies 392 cap-20 values and all 1,560 cap-40 values as simulated.

3 comments
AuthorsOpenReview ↗

Response to W:

Thank you for noting that the original manuscript already distinguished observed rankings from selection gains on held-out search repeats. Your question about what the comparison adds led us to examine a decision the original distinction leaves open. Before seeing any scores for a new model, how much history from the same task is worth consulting? In the revision we hold out an entire model configuration. Section 3.2 gives the recommendation and scoring rules, and Section 4.5 varies the number of historical configurations available while evaluating the choices on the same targets.

Table 6 keeps the target configuration fixed and varies how many previous configurations are available at the time of recommendation. On online bin packing, one historical configuration gave a 59.4% mean probability of recommending a framework ranked first on the target (including ties), compared with 76.9% using all 25. On Erdős, two historical configurations already matched the 96.2% hit rate obtained with all 25. For the Erdős configurations we studied, a small amount of same-task history was enough to make a stable first choice. On bin packing, relying on a single past model more often missed the leader on the new one.

We also checked the quality attained by each recommendation, since a change in rank can correspond to a very small score difference. Across the five tasks, pooling the historical scores gave higher quality on 121 targets, equal quality on 2, and lower quality on 7 relative to the average choice based on one past configuration. Thank you for asking us to spell out the advance over prior comparisons. The revision tests whether past rankings can guide the next choice and gives task-specific evidence for how much confidence to place in that choice. We now frame the contribution in these terms in Section 2.1, the introduction, and the conclusion.

AuthorsOpenReview ↗

Response to Q1:

Thank you for deriving the integer checks from our scoring definitions and testing them against the released run records. By specifying the quantities, threshold, and counts for each task, you gave us a check we could repeat run by run. You found a genuine inconsistency. Under the stated protocol, M is the mean makespan on three flowshop instances, so 3M should recover the sum of three integer makespans. For bin packing, Q is the summed volume lower bound (9,939 bins) divided by the total number of bins used across the five streams, so 9939/Q should recover that integer bin count. We applied both checks to all 624 relevant records across the four frameworks. For flowshop, 3M was more than 0.1 from the nearest integer in 207 of 312 runs. For bin packing, 9939/Q exceeded the same distance in 183 of 312 runs, giving 390/624 (62.5%) in total. The high-precision source table, the run-level scores, and the native endpoint export all gave the same counts. The discrepancies were present before averaging across repeats or rounding for display.

In Table 3 of the revision, we updated the two EoH summaries at the selected configurations. Parentheses give sample standard deviations across three repeats.

EoH result in Table 3SubmittedRevised
Flowshop mean makespan3465.98 (6.39)3466.00 (1.00)
Bin-packing mean excess (%)0.6170 (0.0807)0.6171 (0.146)

Appendix B.1 gives the bin-packing calculation immediately after Table 9. The volume lower bounds sum to 9,939 bins across the five streams. At the precision of the mean quality reported in Table 9, requiring an integer excess-bin count in each run gives 184 excess bins across the three repeats. This yields a mean excess of 0.6171% and a sample standard deviation of 0.146 percentage points. Thank you again for giving us a reproducible check. It led us to correct these entries in Table 3 and show how readers can recover the bin-packing figures from the scoring definition.

AuthorsOpenReview ↗

Response to Q2:

Thank you for checking the provenance labels in the supplement against Appendix D.2, Table 16, and Figure 14. We checked the source labels for all 1,560 rows, recomputed the per-task counts in the original Table 16, and reconstructed the five transition matrices in the original Figure 14. Your reading was correct. Appendix D.2 summarized the depth comparison in the text. Table 16 counted the categories by task, and Figure 14 showed how often the same runs moved between categories. All three used the same data. At cap 20, 1,168 values came from recorded candidate evaluations and 392 were simulated using donor values from the same task and framework. At cap 40, all 1,560 values were simulated. Every transition in Figure 14 therefore used a simulated cap-40 value. The reported 35.8% measured category changes within that scenario.

In the 26-configuration revision, we removed this comparison from Appendix D.2, along with the original Table 16, Figure 14, and the 35.8% claim. Section 4.4 now uses recorded evaluations to answer two questions. Within a run, are high-scoring candidates more concentrated in execution behavior than random subsets of the same size? We observed this in 902 of 1,168 evaluable runs. When an edit changes a program's execution response, how does its quality change? Among the 167 pairs with a changed response and comparable scores, quality improved in 52, stayed the same in 3, and declined in 112. Thank you for tracing the original figures back to their sources. Your check prompted us to withdraw the inference about candidate depth and to specify which evaluation records support these two comparisons in Section 4.4.

Reviewer_27tGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares four LLM-driven program evolution frameworks (EoH, OpenEvolve, ShinkaEvolve, LoongFlow) on five tasks with 26 model configurations and three repeats (1,560 runs) under a common 50M-token threshold. It analyzes endpoint quality, budget trajectories, candidate dispersion measures (D, L, C), and program-level controls. Its main conclusion is that framework advantages are task-specific and depend on the budget.

Critical strengths

The study is large (1,560 runs) and carefully controlled, with shared evaluators and budgets. The authors also report the limits of their analyses openly. However, these strengths do not compensate for the limited new findings.

Critical weaknesses
  1. Limited new knowledge. The main conclusions are mostly expected. That framework advantages depend on the task and the budget is already visible in Figure 1, and prior work that the paper cites reports task-dependent advantages (Ellis & Castro, 2026) and competitive simple baselines (Gideoni et al., 2026). The paper does not turn its observations into a method, a mechanism, or practical guidance beyond "choose by task and budget".

  2. The diagnostics do not explain the gaps. The dispersion measures (D, L, C), source-pattern tags, and program controls do not explain why one framework outperforms another.

  3. Misleading presentation and weak evaluation. Section 3.3 describes "new-instance experiments", but Appendix H states that all these responses are simulated.

[1] Ellis & Castro, Don't Gamble, GAMBLe: An Analytical Framework for AI-Driven Research Systems, arXiv 2026.

[2] Gideoni et al., Simple Baselines Are Competitive with Code Evolution, ICLR 2026 Workshop on Recursive Self-Improvement.

Questions for authors

Q1. Beyond "choose by task and budget", what practical guidance or mechanism does the study provide? For example, which framework components cause the task-specific advantages?

Q2. Do you have results from real evaluations of the evolved programs on new instances, instead of the simulation in Appendix H?

3 comments
AuthorsOpenReview ↗

Response to W:

Thank you for recognizing the effort to compare four frameworks under shared evaluation conditions. As you point out, even 1,560 runs leave a fair question: what can the reader learn from these comparisons? We reconsidered how we framed the contribution and what evidence supported each claim.

W1. What does the comparison add? The studies you cited helped us see where the original framing put too much weight on rankings changing with the task or model. Section 2.1 now discusses the generator–search pairings studied by Ellis and Castro and the competitive simple baselines reported by Gideoni et al. Section 4.1 compares IID and GFS with all four evolutionary frameworks. On the selected TSP configuration, for example, IID and three of the four frameworks achieved zero mean tour gap.

Building on those comparisons, we ask a practical question: when several models have already been evaluated on a task, how much does using their combined history improve on choosing the winner from a single past evaluation? Section 4.5 holds out each of 26 model configurations on each of five tasks. For each target, one rule selects the framework with the highest mean score across the other 25 configurations. The other uses one historical configuration, and we average over all 25 possible choices. Both rules choose among the same four frameworks and are scored on the same target results. This lets us measure what pooling the historical scores adds to the choice.

Across 130 targets, pooling gave higher quality on 121, the same quality on 2, and lower quality on 7. We can also measure the size of the difference. On TSP, for example, the mean quality gap between the selected framework and the best framework on each target fell from 0.0136 to 0.00356. The first result tells a practitioner how often consulting more history led to a better choice in this comparison; the second shows how much closer that choice came to the best available outcome on one task. This is the empirical contribution we have added to the discussion of prior framework comparisons.

W2. What do the diagnostics explain? You identified a gap in our reasoning. Differences in candidate behavior alone cannot tell us which framework component produced an endpoint advantage. Section 3.2 defines what D, L, and C each measure, Section 4.4 reports where concentration could be evaluated, and Appendix C.3 compares 241 pairs of original and edited programs with comparable scores under shared inputs and evaluation conditions.

Appendix C.4 traces one placement to the code that made it. For an item of size 48, a bin with 52 units remaining received a score of −1.022957. A bin with 100 units remaining kept the default score of −1. The driver takes the maximum returned score, so it chose the latter bin. That explains this placement. The additional 208 bins across five complete streams measure the combined effect of replacing the scoring formula and its return logic. The revision distinguishes the explanation for one decision from the measured effect of the full replacement.

W3. Where did the original new-instance results come from? Thank you for checking Section 3.3 against Appendix H. Your reading was correct. At submission, these responses came from simulation. Calling them “new-instance experiments” in the main text gave readers reason to think we had executed the evolved programs on those instances. We removed those simulated results from the revision. Section 3.1 specifies the fixed inputs used in the main experiments, and Sections 3.2 and 4.5 identify the model configuration as the unit held out when evaluating recommendations. Our response to Q2 gives the new-input executions carried out during revision and the two other evaluations reported alongside them. We appreciate the care you took to trace these results to their source.

AuthorsOpenReview ↗

Response to Q1:

Thank you for asking which framework a practitioner should try first and which design choices might account for its advantage. On a familiar task with results from several model configurations, Section 4.5 supports starting with the framework that had the best historical mean score. When each target configuration was withheld until after the choice, this rule selected a first-ranked framework on 109 of 130 targets, including ties. The historical means selected ShinkaEvolve for circle packing and online bin packing, LoongFlow for Erdős, OpenEvolve for flowshop, and EoH for TSP. Each task gave the same recommendation whichever configuration we held out. One exception to the resulting prediction illustrates how we would use it. On TSP with non-thinking Qwen3.5-397B-A17B-FP8, the recommended framework, EoH, scored 0.935317, while ShinkaEvolve scored 1. Historical results can guide the first trial, and evaluation on the target model still matters.

On component attribution, all four frameworks retain their native search loops, so candidate selection, program editing, and population management can vary together. A controlled comparison in Appendix C.4 lets us answer a narrower design question. On the same five item streams, each of 12 evolved programs handled the first half of a stream before best-fit took over. Compared with using best-fit throughout, five programs still saved bins and seven tied. Across the 60 program–stream combinations, they saved 184 bins in total. The remaining items and the second-half policy were identical, so the placements made in the first half affected the final count through the bins and capacities left at handover. Attributing an overall framework gap to a particular search component would require changing that component independently. We appreciate your pressing this distinction. It helped us state the program-level mechanism that the comparison actually tests.

AuthorsOpenReview ↗

Response to Q2:

Yes. Thank you for asking specifically for executions on new instances. During revision we kept the selected evolved programs fixed and ran them on new flowshop, online bin-packing, and TSP inputs. Their outputs passed the validity checks for all three tasks. This test in Appendix C.5 asks whether the programs produce valid solutions on new inputs. We also measured how solution quality changes when an input is perturbed, and what scores frozen programs obtain when rerun on their original instances. The three evaluations use different inputs and report different outcomes.

EvaluationWhat we evaluatedLocation in the revision
Program execution on new inputsWe ran the selected programs on new scheduling instances, item streams, and TSP instances, and checked the validity of the solutions they produced.Appendix C.5, p. 27
Sensitivity of returned solutions to input perturbationsOf 1,560 search runs, 1,559 returned a solution. Starting from each returned solution, we evaluated quality under three input perturbations, giving 4,677 solution–scenario evaluations. For example, on TSP we retained the tours and perturbed the city coordinates. The mean quality changes were −0.40926, −0.88453, and −0.97083 for the three scenarios.Appendix C.5, Table 20, p. 27
Repeat execution on the original instancesWe performed 314 executions of 290 distinct frozen programs, recording the native task score and quality for each execution.Appendix C.6, p. 27

We appreciate your asking how these results were obtained. Readers can now locate the tests in which a program ran on a new input, and separately examine the sensitivity of a returned solution and the results of rerunning a program on its original instances.

Reviewer_wfthOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares four frameworks for iteratively generating and evaluating programs with LLMs (EoH, OpenEvolve, ShinkaEvolve, and LoongFlow) on five tasks (circle packing, Erdős minimum overlap, flowshop scheduling, online bin packing, and the traveling salesperson problem). Across 26 model/reasoning configurations and three repeats, it reports 1,560 runs. It finds recurring task-specific winners and examines framework selection, budget effects, and generated-program behavior.

Critical strengths
  1. Several task-specific winners recur across many models. The broad comparison provides useful benchmark guidance; for example, LoongFlow leads Erdős and OpenEvolve leads flowshop in 25 of 26 configurations.
Critical weaknesses
  1. The main findings provide limited new insight. It is not especially surprising that the best framework depends on the task and budget. The paper gives limited explanation of why, or how to use these findings to design better search methods.

  2. The results give little guidance for a new problem. Benchmark rankings do not establish which framework to choose elsewhere. For frontier discovery, the budget analysis also leaves unclear when to keep searching, switch frameworks, or combine them.

Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Response to W:

Thank you for drawing attention to the recurring leads of LoongFlow on Erdős and OpenEvolve on flowshop, and for asking what someone could do with that observation. Looking back at the original manuscript, we could identify a winner once all the scores were in. A practitioner switching models has to choose a framework before seeing the new model's scores. Your review prompted us to test that choice directly.

Section 3.2 turns this into a held-out test. On each task, we set aside one entire model configuration, choose a framework using mean scores on the other 25, and then check that choice against all four frameworks on the target. This separates two questions. Would changing the available history change the recommendation? And even if the recommendation is stable, does the chosen framework do well on the target? In Section 4.5, each task gave the same recommendation in all 26 holdouts: for example, always OpenEvolve for flowshop and always ShinkaEvolve for online bin packing. Across the 130 targets, the recommended framework ranked first outright on 96 and tied for first on 13. A different framework led on the other 21. For someone who already has results on the same task, this gives a concrete first trial when switching models: start with the framework with the best historical mean. The holdouts also show how often that choice agreed with the best framework on the target configurations we studied. We revised the introduction and conclusion to make this finding explicit.

Your question about search design also led us to look more closely at what an evolved program actually does. In Section 4.4 and Appendix C.4, we trace one OpenEvolve bin-packing program from its internal choice to the scores it returns. The driver can select a different bin from the one the function prefers internally. Keeping the driver and five item streams fixed, replacing the scoring formula and its return logic increased the total from 10,127 to 10,335 bins. This suggests a concrete diagnostic: record the score vector passed to the driver, then check how the resulting placements affect the final bin count.

A wholly new task offers no same-task scores on which to base that first choice. For a search already in progress, our budget curves follow each framework as it continues its own search. Deciding when to switch or combine frameworks would require comparing those actions from the same intermediate state with the same remaining budget. We appreciate your raising both situations. It helped us specify exactly where cross-model stability can guide a decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.