- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares four LLM-driven program evolution frameworks across five tasks and 26 model/reasoning configurations, reporting 1,560 runs under a common 50M-token request-start threshold. It examines task-dependent framework advantages, model effects, and budget-dependent progress, supplemented by execution-profile analysis and selected program-level controls.
Critical strengths
- The comparison covers multiple frameworks, tasks, and model configurations, providing a broad view of performance under the evaluated settings.
- The paper distinguishes observed rankings from selection gains on held-out search repeats. It also explicitly acknowledges the limits of fixed-input evaluation, reconstructed budget coordinates, and program-level attribution.
Critical weaknesses
The paper's additional insight beyond prior work remains unclear.
The main conclusions are that framework advantages depend on tasks, models, and budgets, and that response dispersion alone does not identify the best-performing framework. These findings are broadly consistent with reasonable expectations. Section 2 discusses studies addressing related factors, but the paper does not clearly identify which specific findings materially extend that understanding or provide additional guidance for framework selection. For a paper centered on comparative analysis, this incremental contribution needs clearer articulation.
Questions for authors
-
The stated protocol implies that for Flowshop and for bin packing should be integers. Across the four frameworks, 207+183=390 of the 624 run-level records (approximately 62.5%) yield values more than 0.1 away from the nearest integer. Could the authors explain these discrepancies, including any additional aggregation, precision handling, or evaluator differences?
-
Could the authors clarify which results in Appendix D.2, Table 16, and Figure 14 come from actual candidate evaluations and which come from simulation? The supplementary provenance identifies 392 cap-20 values and all 1,560 cap-40 values as simulated.
3 comments
Response to Q1:
Thank you for deriving the integer checks from our scoring definitions and testing them against the released run records. By specifying the quantities, threshold, and counts for each task, you gave us a check we could repeat run by run. You found a genuine inconsistency. Under the stated protocol, M is the mean makespan on three flowshop instances, so 3M should recover the sum of three integer makespans. For bin packing, Q is the summed volume lower bound (9,939 bins) divided by the total number of bins used across the five streams, so 9939/Q should recover that integer bin count. We applied both checks to all 624 relevant records across the four frameworks. For flowshop, 3M was more than 0.1 from the nearest integer in 207 of 312 runs. For bin packing, 9939/Q exceeded the same distance in 183 of 312 runs, giving 390/624 (62.5%) in total. The high-precision source table, the run-level scores, and the native endpoint export all gave the same counts. The discrepancies were present before averaging across repeats or rounding for display.
In Table 3 of the revision, we updated the two EoH summaries at the selected configurations. Parentheses give sample standard deviations across three repeats.
| EoH result in Table 3 | Submitted | Revised |
|---|---|---|
| Flowshop mean makespan | 3465.98 (6.39) | 3466.00 (1.00) |
| Bin-packing mean excess (%) | 0.6170 (0.0807) | 0.6171 (0.146) |
Appendix B.1 gives the bin-packing calculation immediately after Table 9. The volume lower bounds sum to 9,939 bins across the five streams. At the precision of the mean quality reported in Table 9, requiring an integer excess-bin count in each run gives 184 excess bins across the three repeats. This yields a mean excess of 0.6171% and a sample standard deviation of 0.146 percentage points. Thank you again for giving us a reproducible check. It led us to correct these entries in Table 3 and show how readers can recover the bin-packing figures from the scoring definition.
Response to Q2:
Thank you for checking the provenance labels in the supplement against Appendix D.2, Table 16, and Figure 14. We checked the source labels for all 1,560 rows, recomputed the per-task counts in the original Table 16, and reconstructed the five transition matrices in the original Figure 14. Your reading was correct. Appendix D.2 summarized the depth comparison in the text. Table 16 counted the categories by task, and Figure 14 showed how often the same runs moved between categories. All three used the same data. At cap 20, 1,168 values came from recorded candidate evaluations and 392 were simulated using donor values from the same task and framework. At cap 40, all 1,560 values were simulated. Every transition in Figure 14 therefore used a simulated cap-40 value. The reported 35.8% measured category changes within that scenario.
In the 26-configuration revision, we removed this comparison from Appendix D.2, along with the original Table 16, Figure 14, and the 35.8% claim. Section 4.4 now uses recorded evaluations to answer two questions. Within a run, are high-scoring candidates more concentrated in execution behavior than random subsets of the same size? We observed this in 902 of 1,168 evaluable runs. When an edit changes a program's execution response, how does its quality change? Among the 167 pairs with a changed response and comparable scores, quality improved in 52, stayed the same in 3, and declined in 112. Thank you for tracing the original figures back to their sources. Your check prompted us to withdraw the inference about candidate depth and to specify which evaluation records support these two comparisons in Section 4.4.
Response to W:
Thank you for noting that the original manuscript already distinguished observed rankings from selection gains on held-out search repeats. Your question about what the comparison adds led us to examine a decision the original distinction leaves open. Before seeing any scores for a new model, how much history from the same task is worth consulting? In the revision we hold out an entire model configuration. Section 3.2 gives the recommendation and scoring rules, and Section 4.5 varies the number of historical configurations available while evaluating the choices on the same targets.
Table 6 keeps the target configuration fixed and varies how many previous configurations are available at the time of recommendation. On online bin packing, one historical configuration gave a 59.4% mean probability of recommending a framework ranked first on the target (including ties), compared with 76.9% using all 25. On Erdős, two historical configurations already matched the 96.2% hit rate obtained with all 25. For the Erdős configurations we studied, a small amount of same-task history was enough to make a stable first choice. On bin packing, relying on a single past model more often missed the leader on the new one.
We also checked the quality attained by each recommendation, since a change in rank can correspond to a very small score difference. Across the five tasks, pooling the historical scores gave higher quality on 121 targets, equal quality on 2, and lower quality on 7 relative to the average choice based on one past configuration. Thank you for asking us to spell out the advance over prior comparisons. The revision tests whether past rankings can guide the next choice and gives task-specific evidence for how much confidence to place in that choice. We now frame the contribution in these terms in Section 2.1, the introduction, and the conclusion.