acceptodds
Under review as a conference paper at ICLR 2027

FrugalEvo: Towards Cost-Aware LLM-Guided Program Evolution

Abstract

LLM-guided evolutionary methods, such as AlphaEvolve, have emerged as powerful approaches for challenging computational optimization problems, such as circle packing. However, prior work typically optimizes performance gain over a fixed number of iterations. We argue that practical optimization should maximize gain per unit cost. To this end, we propose FrugalEvo, a cost-aware evolutionary framework where a stronger, higher-cost LLM explores solution strategies, and a cheaper LLM implements them and iteratively refines the resulting code. We also design a cache-efficient evolution process, where our harness and prompts maximize the sharing of prefixes across different evolution steps, to improve cache reuse. To measure solution quality throughout a fixed cost budget, we introduce Budget-Aware Area Under the Curve (BA-AUC), defined as the area under the best-so-far evaluation score curve over cumulative LLM cost, up to the budget. Across 10 mathematical and systems optimization tasks, FrugalEvo matches or surpasses state-of-the-art baselines, including OpenEvolve, ShinkaEvolve, AdaEvolve, and EvoX, in final solution quality and achieves higher BA-AUC on 9 tasks. It also achieves higher average performance than these baselines on 10 algorithmic optimization tasks from ALE-Bench-Lite. Notably, on circle packing, FrugalEvo achieves new state-of-the-art performance with GPT-5.6 Terra and Luna for only 1.68 USD and with GLM-5.3 and its Flash variant for only 0.55 USD, matching or surpassing all baselines, including multi-agent methods such as CORAL and SwarmResearch, which cost approximately 50 USD on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evolution, Scientific Discovery, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bU8dOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper adopts a division of labor between a stronger, more expensive LLM for strategy exploration and a cheaper LLM for code implementation, aiming to improve performance gain per unit cost. It further optimizes prompt prefixes to enable cache-efficient generation. Experiments on 20 tasks show that FrugalEvo achieves comparable or better performance than four existing baselines while improving Budget-Aware AUC (BA-AUC), which measures the quality of the best-so-far solution over cumulative LLM expenditure and thus evaluates optimization performance under cost constraints.

Critical strengths
  1. This paper uses a relatively simple division of labor between LLMs to reduce inference costs. This is a timely and practically relevant direction, as the high cost of LLM usage remains a significant barrier for researchers with limited computational budgets.

  2. The framework and pipeline are clearly described with an algorithm box, a clean high-level schematic, and a precise definition of BA-AUC. The rationale behind each component (strategy explorer, solution generator, search memory) is easy to follow.

Critical weaknesses
  1. Limited methodological novelty and overextended claims regarding "cost-awareness" and caching. (1) The "cost-aware" logic lacks dynamic adaptation—it merely enforces a static, predefined division of labor (strong LLM for strategy exploration, cheap LLM for code implementation) without any dynamic, online cost-performance feedback or adaptive routing. (2) Prompt caching is an existing infrastructure capability provided by commercial API vendors rather than an algorithmic contribution of this paper; the authors merely apply standard prompt engineering by ordering static instructions first to exploit vendor prefix-matching rules. (3) The reliance on vendor-specific pricing and caching discount schemes raises significant generalizability concerns, as the reported cost advantages depend heavily on current commercial API pricing models and may diminish as base inference costs continue to drop.

  2. Confounded Baseline Evaluation and Missing Sensitivity Analysis. (1) Unfair Baseline Comparisons: Baselines are restricted to a single high-cost model, whereas FrugalEvo uniquely benefits from a dual-model cascade and prompt caching. Equipping baselines with a similar dual-model setup is necessary to isolate true algorithmic search gains from API price arbitrage. (2) Missing Hyperparameter Sensitivity Analysis: The paper lacks sensitivity analysis for crucial search hyperparameters—including strategy count KK, refinement cap MM, and cold-start limit N0N_0—leaving framework robustness and hyperparameter tuning across domains unverified.

Questions for authors
  1. How much of FrugalEvo's improvement in BA-AUC and optimization performance can be attributed separately to the evolution strategy, heterogeneous model collaboration, and cache-efficient prompt construction? In particular, could the authors provide a caching-on versus caching-off comparison under otherwise identical settings?

  2. Could representative baselines, such as OpenEvolve or EvoX, be evaluated with the same two-model cascade and comparable prompt-caching support under matched cost budgets? This comparison would help determine whether FrugalEvo's advantage comes from its evolution strategy or primarily from access to heterogeneous models and lower-cost inference. Results from such a controlled comparison could materially affect my assessment of the paper's central contribution.

  3. What makes the proposed framework cost-aware at the search level? Beyond evaluating methods using BA-AUC under a fixed cost budget, does FrugalEvo adapt model-call allocation, strategy exploration, or refinement decisions in response to observed cost-performance feedback? If so, could the authors clarify the mechanism and provide evidence of its effect? If the allocation is largely predefined, could the authors clarify how the claimed contribution differs from a fixed two-model cascade combined with cache-efficient prompting? This clarification would help establish the methodological scope and novelty of the work.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bU8d (Part 1/2)

We thank Reviewer bU8d for finding our work explores a timely and practically relevant direction and for appreciating our framework and pipeline are clearly described. We are also glad that the reviewer finds the rationale behind each component in our work is easy to follow. Below, we address the key questions raised and hope our responses resolve the concerns.


W1: Limited methodological novelty and overextended claims regarding "cost-awareness" and caching.

Thanks for pointing out these concerns. We address them separately below.

(1) The “cost-aware” logic lacks dynamic adaptation. We agree that our method does not perform adaptive model routing. The assignment of strategy exploration to a stronger, more expensive model and code implementation to a cheaper model is fixed. This design is motivated by the observation that, in computational optimization tasks, algorithmic structure and the choice of optimization operators can substantially affect solution quality. We therefore allocate the stronger model to explore the design strategies, while using the cheaper model to implement and refine them. Our intended contribution is this strategy–implementation decomposition and its empirical effectiveness under a fixed budget. We will revise the description to make this scope explicit.

(2) Role of prompt caching. We agree that prompt caching is a vendor-provided capability, rather than an algorithmic contribution of our work. Our discussion was intended to highlight the practical importance of cache reuse in evolutionary search and how prompt organization can exploit this capability to improve cost efficiency. In the revision, we will clarify that this is an implementation-level optimization.

(3) Pricing dependence and generalizability. We agree that the magnitude of cost savings depends on model prices and caching discounts. Our primary focus is on how to combine two models with different costs and capabilities to obtain better solutions under a fixed cost budget. We will explicitly acknowledge this as a limitation in the revision.

Overall, our work aims to discuss how evolutionary search can be designed to maximize performance gains per unit cost rather than per iteration. This perspective has received limited attention in prior work, yet it is important in practice.


W2&Q2: Confounded Baseline Evaluation and Missing Sensitivity Analysis.

Thanks for this comment. We address each concern below.

(1) Unfair Baseline Comparisons. We agree that adding baselines with a similar dual-model setup is necessary. Table C and Table D present results of model-mixing baselines.

  1. Table C: Results on Signal Processing with model-mixing baselines. Both FrugalEvo and all baselines use the expensive and cheap models together (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)0.777912 ± 0.01450.7891160.784035 ± 0.00340.786413
    Baselines (Terra+Luna / GLM5.3+Flash)
    OpenEvolve0.545894 ± 0.00540.5513150.715171 ± 0.00260.717748
    ShinkaEvolve0.562811 ± 0.00720.5699660.728831 ± 0.00450.733307
    AdaEvolve0.726819 ± 0.01010.7369240.760954 ± 0.00990.770806
    EvoX0.547480 ± 0.02160.5691220.640309 ± 0.08510.725454
  2. Table D: Results on Circle Packing with model-mixing baselines. The model settings and budgets are the same as in Table C.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)2.635989 ± 0.00002.6359962.635422 ± 0.00082.635990
    Baselines (Terra+Luna / GLM5.3+Flash)
    OpenEvolve2.588702 ± 0.06692.6359832.627872 ± 0.00642.634292
    ShinkaEvolve2.585999 ± 0.06312.6305882.599917 ± 0.03602.635953
    AdaEvolve2.630226 ± 0.00022.6303812.630599 ± 0.00542.635977
    EvoX2.585497 ± 0.06232.6295722.595936 ± 0.01582.611689

As shown in Table C and Table D, FrugalEvo surpasses all the baselines on both tasks given the same budget. This indicates that our model collaboration design is more efficient on these mathematical optimization tasks.

AuthorsOpenReview ↗

Response to Reviewer bU8d (Part 2/2)

(2) Sensitivity Analysis. We agree that adding sensitivity analysis is important. We have conducted a sensitivity analysis of strategy count , refinement cap , and cold-start limit on the Signal Processing task using GPT-5.6. The results are presented in Table E.

  1. Table E: Sensitivity analysis of strategy count (K), refinement cap (M), and cold-start limit (C) on Signal Processing using GPT-5.6 under a USD 2 budget per run. We report mean ± standard deviation and best scores across 3 runs.
ConfigurationAvg ± StdBest
FrugalEvo (K=3, M=3, C=3)0.777912 ± 0.01450.789116
K=1, M=3, C=30.734899 ± 0.02630.761151
K=5, M=3, C=30.758914 ± 0.02040.779275
K=3, M=1, C=30.774967 ± 0.00830.783267
K=3, M=5, C=30.746496 ± 0.02460.771093
K=3, M=3, C=10.718922 ± 0.01490.733778
K=3, M=3, C=50.776601 ± 0.03140.807969

As shown in Table E, the default configuration achieves the highest average score among the evaluated settings. Increasing C to 5 yields a similar average and the highest best score, but also greater variability across different runs. Moreover, other parameter changes lead to lower average scores.


Q1: How much of FrugalEvo's improvement in BA-AUC and optimization performance can be attributed separately to the evolution strategy, heterogeneous model collaboration, and cache-efficient prompt construction? Could the authors provide a caching-on versus caching-off comparison under otherwise identical settings?

Thank you for raising this question.

  • Performance improvement attribution.

The performance improvement of FrugalEvo is most likely attributed to our model collaboration design for the computational optimization tasks. As discussed in our response to W1, this design is motivated by our observations about the characteristics of these optimization tasks.

  • Caching-on and caching-off comparison.

Our cache-efficient prompt construction is intended primarily to reduce API costs. We analyzed Circle Packing runs of FrugalEvo using GLM 5.3 and its Flash variant across three runs, each with a 1 USD budget.

To isolate the caching discount, we kept the recorded prompts, model calls, generated programs, and evaluation results identical. For caching-off, we recalculated costs by charging all input tokens at the regular input rate, while keeping output pricing unchanged.

The average cost across three runs is 0.776558 USD with caching enabled and 0.837117 USD with caching disabled. Caching reduces the average cost by 0.060559 USD, corresponding to an aggregate cost reduction of 7.23%.


Q3: What makes the proposed framework cost-aware at the search level?

Thanks for this question. FrugalEvo is designed to improve solution quality under a fixed cost budget; it does not dynamically allocate model calls based on observed cost–performance feedback.

The assignment of strategy exploration to a stronger model and implementation to a cheaper model is predefined, motivated by our observations about the characteristics of computational optimization tasks.

Thank you for the additional experiments and clarifications. I appreciate the efforts to introduce model-mixing baselines and conduct hyperparameter sensitivity analysis. These additions improve the experimental evidence and partially address my concerns regarding baseline fairness and robustness.

However, my primary concern regarding methodological novelty remains unresolved. As the authors explicitly acknowledge, FrugalEvo does not dynamically adapt model allocation, strategy exploration, or refinement decisions based on observed cost-performance feedback. Instead, it relies on a predefined division of labor between a stronger model for strategy exploration and a cheaper model for code implementation. While this is a reasonable and practically useful design choice, the underlying idea is relatively straightforward and, in my view, represents more of a lightweight engineering heuristic than a substantial methodological contribution.

Overall, the additional experiments provide stronger empirical support for the practical effectiveness of FrugalEvo, but they do not fundamentally change my assessment of its methodological contribution. I acknowledge its practical value; however, I remain unconvinced that the proposed framework offers sufficient methodological novelty for acceptance at ICLR.

Therefore, I maintain my original rating of 2 (Weak Reject).

Thank you for acknowledging our additional experiments and for the thoughtful follow-up. We agree that FrugalEvo uses a fixed division of labor rather than dynamically adapting model allocation, and we would like to briefly clarify why we view this design as more than an engineering heuristic.

First, the division of labor in FrugalEvo is derived from an empirical insight into computational optimization problems: large performance gaps between solutions mainly stem from whether the right operator or algorithm structure is discovered, while refinement within the same operator yields only marginal gains. FrugalEvo is built directly on this insight, allocating the strong model to strategy exploration and the cheap model to repeated refinement. We believe that identifying where model capability matters most in the evolutionary search process for computational optimization tasks is itself a methodological contribution.

Second, the model-mixing baselines provide evidence that the gains come from this specific design rather than from simply combining models. These baselines use exactly the same models and budget as FrugalEvo, yet still underperform it (Tables C and D). This indicates that how capability is allocated across the search process, rather than model mixing per se, is the key to FrugalEvo’s effectiveness.

Finally, we view the simplicity of FrugalEvo as a strength: it can be easily integrated into existing evolutionary frameworks. We agree that dynamic model allocation is a promising direction, and we will discuss it as future work in the revised version.

We sincerely appreciate your time and constructive feedback throughout the discussion!

Reviewer_6CffOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FrugalEvo aims to improve LLM-guided program evolution under a fixed API budget. It separates strategy design from implementation. A stronger, more expensive model proposes optimization strategies. A cheaper model implements them, repairs errors, and refines programs using evaluation feedback. The framework also includes a cheap-model cold start, strategy ranking through initial implementations, and search-history management. Its prompts are organized to encourage prefix-cache reuse.

The authors introduce BA-AUC, which measures best-so-far solution quality over cumulative API expenditure. They evaluate FrugalEvo on five mathematical tasks, five systems tasks, and ten algorithmic tasks from ALE-Bench-Lite. The experiments use two pairs of strong and inexpensive models. FrugalEvo reports competitive or better final performance and BA-AUC on most mathematical and systems tasks. It also achieves the highest average private score on ALE-Bench-Lite.

Critical strengths
  1. The paper tackles test-time evolution, an important research direction, while also incorporating cost-awareness, which is a valuable and practical consideration.
  2. The experiments cover diverse tasks and show promising gains.
  3. The writing is generally easy to follow
Critical weaknesses

FrugalEvo combines strong and inexpensive models, while all four primary baselines use only the expensive model. This is a valid comparison of complete systems under a fixed API budget. It does not establish how much the strategy–implementation decomposition contributes. Cheaper calls alone may allow more search attempts and explain part of the improvement.

Table 3 provides partial evidence through the Terra+Terra and Luna+Luna ablations. These comparisons remain within FrugalEvo and cover only two mathematical tasks. Baselines using the inexpensive model would help test whether the framework adds value beyond lower inference costs. Simple model-mixing or switching baselines would further test the proposed assignment of roles. This is my main concern with the experimental evidence.

Minor: the method that routes between strong (expensive) and weak (cheap) models is not new.

Questions for authors

Could the authors disentangle the performance gains attributable to using a cheaper yet effective model (e.g., GPT-5.6-Luna) from those resulting from the proposed cost-aware selection strategy? In particular, how would the baselines perform if they all used the same cheaper model?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6Cff (Part 1/2)

We thank Reviewer 6Cff for recognizing that our work tackles an important research problem and for finding that our experiments show promising gains across various tasks. We are also glad that the reviewer finds our writing is easy to follow. Below, we address all the questions raised in the review and hope our responses adequately resolve the concerns.


W1&Q1: Cheaper calls alone may allow more search attempts and explain part of the improvement.

Thanks for this suggestion. We agree that including baselines that use only cheaper models would better demonstrate the effectiveness of our method. We have added these baselines on Circle Packing and Signal Processing tasks, as shown in Table A and Table B below.

  1. Table A: Results on Signal Processing with cheaper-model-only baselines. FrugalEvo uses both the strong and cheap models (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3), while all baselines use only the cheap model (Luna or Flash). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)0.777912 ± 0.01450.7891160.784035 ± 0.00340.786413
    Baselines (Luna / Flash only)
    OpenEvolve0.633766 ± 0.02780.6596580.688770 ± 0.04890.723374
    ShinkaEvolve0.556792 ± 0.09730.6690830.689766 ± 0.06000.732171
    AdaEvolve0.752368 ± 0.00560.7587810.737765 ± 0.02240.753594
    EvoX0.546776 ± 0.02370.5720480.573644 ± 0.02970.594673
  2. Table B: Results on Circle Packing with cheaper-model-only baselines. The model settings are the same as in Table A.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)2.635989 ± 0.00002.6359962.635422 ± 0.00082.635990
    Baselines (Luna / Flash only)
    OpenEvolve2.632631 ± 0.00172.6342922.622509 ± 0.00792.630367
    ShinkaEvolve2.635983 ± 0.00002.6359832.622527 ± 0.00752.630048
    AdaEvolve2.626094 ± 0.00992.6359832.626615 ± 0.00452.631105
    EvoX2.630705 ± 0.00032.6309722.633538 ± 0.00242.635983

As shown in Table A and Table B, under the same budget, our method outperforms all the baselines on both tasks, even though the baselines use only cheaper models and can thus evolve for more iterations. The performance gap is particularly clear on the signal processing task.

AuthorsOpenReview ↗

Response to Reviewer 6Cff (Part 2/2)

W2: Simple model-mixing or switching baselines would further test the proposed assignment of roles.

Thanks for this thoughtful comment. We agree that adding simple model-mixing baselines would enable a more in-depth comparison of our method’s performance. These results are presented in Table C and Table D.

For model mixing, OpenEvolve selects the cheap model with an 80% probability and the expensive model with a 20% probability for candidate generation. EvoX, AdaEvolve, and ShinkaEvolve use the cheap model for candidate generation and the expensive model to provide high-level guidance for the search.

  1. Table C: Results on Signal Processing with model-mixing baselines. Both FrugalEvo and all baselines use the expensive and cheap models together (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)0.777912 ± 0.01450.7891160.784035 ± 0.00340.786413
    Baselines (Terra+Luna / GLM5.3+Flash)
    OpenEvolve0.545894 ± 0.00540.5513150.715171 ± 0.00260.717748
    ShinkaEvolve0.562811 ± 0.00720.5699660.728831 ± 0.00450.733307
    AdaEvolve0.726819 ± 0.01010.7369240.760954 ± 0.00990.770806
    EvoX0.547480 ± 0.02160.5691220.640309 ± 0.08510.725454
  2. Table D: Results on Circle Packing with model-mixing baselines. The model settings and budgets are the same as in Table C.

    MethodGPT5.6 AvgGPT5.6 BestGLM5.3 AvgGLM5.3 Best
    FrugalEvo (Terra+Luna / GLM5.3+Flash)2.635989 ± 0.00002.6359962.635422 ± 0.00082.635990
    Baselines (Terra+Luna / GLM5.3+Flash)
    OpenEvolve2.588702 ± 0.06692.6359832.627872 ± 0.00642.634292
    ShinkaEvolve2.585999 ± 0.06312.6305882.599917 ± 0.03602.635953
    AdaEvolve2.630226 ± 0.00022.6303812.630599 ± 0.00542.635977
    EvoX2.585497 ± 0.06232.6295722.595936 ± 0.01582.611689

As shown in Table C and Table D, our FrugalEvo surpasses all the baselines on both tasks given the same budget. This indicates that our model collaboration design is more efficient on these mathematical optimization tasks.


Minor: the method that routes between strong (expensive) and weak (cheap) models is not new.

Thank you for raising this concern. We agree that combining strong and weak models is not new. However, rather than simply pairing two models, our design is motivated by the characteristics of computational optimization problems.

Before designing FrugalEvo, we analyzed solutions to several such problems and found that large score gaps between solutions usually come from whether a particular optimization operator is used. In other words, finding a substantially better solution depends mainly on discovering the right operator. Based on this observation, FrugalEvo assigns the expensive strong model to operator search and algorithm structure design, and uses the cheap model for repeated refinement. This division of labor allows FrugalEvo to spend the budget where it has the greatest impact.

Reviewer_6CffOpenReview ↗

Response to Author's rebuttal

Thank you for the effort. I raised my score because the comparison is fairer.

Thank you for the positive feedback and for raising your score! We are glad that our response has addressed your concerns.

Reviewer_xoxaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

FrugalEvo is an LLM-guided program-evolution framework built for a fixed dollar budget. An expensive model proposes K design strategies for the current best program; a cheap model implements each strategy and refines it for up to M attempts; a cold-start phase lets the cheap model improve the initial program first; prompts are laid out so that shared prefixes can be served from the API's prompt cache (Sec. 2.2, Fig. 2, Alg. 1). The paper also proposes BA-AUC, the area under the best-so-far score curve plotted against cumulative LLM cost, up to the budget (Eq. 4). Experiments cover 5 mathematical tasks, 5 systems tasks from ADRS-Bench and 10 ALE-Bench-Lite tasks. Two configurations are used: GPT-5.6 Terra with GPT-5.6 Luna, and GLM-5.3 with GLM-5.3-Flash; ALE-Bench-Lite is run with the GPT configuration only. Baselines are OpenEvolve, ShinkaEvolve, AdaEvolve and EvoX, each run with the expensive model only, at budgets of $2 / $1 (GPT) and $1 / $0.5 (GLM), three runs each (Sec. 3.1; Tables 1–2 captions). FrugalEvo reports the best mean BA-AUC on 9 of 10 math and systems tasks, best or comparable final scores, the best average on ALE-Bench-Lite (Fig. 4), and a circle-packing sum of radii of 2.635996 reached for $1.68 (Tables 1–2, Fig. 3). An ablation compares the two-model split with single-model variants and removes the cold start and the sequential feedback (Table 3).

Critical strengths
  • The question is the right one: iteration counts hide large cost differences, and plotting score against cost (Fig. 5) is more informative than score against iterations.
  • The experimental breadth is good: 20 tasks across three families, two model families, four current baselines, three runs per cell, with mean and standard deviation reported (Tables 1–2).
  • The ablation in Table 3 is informative. It shows that the two-model split beats either model used alone inside the same framework (circle packing 2.635989 vs 2.634565 for the cheap model alone; Signal Processing 0.77791 vs 0.74865).
  • The prompts (App. D), the algorithm (Alg. 1), the benchmark descriptions (App. B) and the price table (App. C) are given, and the paper promises code release.
  • The qualitative descriptions of discovered solutions (Sec. 3.2) make the results easier to interpret.
Critical weaknesses
  1. The comparison does not separate the framework from the model mix, and the paper's own ablation suggests the cheap model is doing much of the work. FrugalEvo uses an expensive model plus a cheap one; every baseline uses the expensive model alone (Sec. 3.1). The cheap model costs one tenth per token (App. C), and Fig. 1c shows FrugalEvo running about twice as many iterations per dollar. Table 3 shows what the framework does with one model: with Terra in both roles, FrugalEvo scores 2.632278 on circle packing and 0.68534 on Signal Processing, which places it in the middle of the Terra-only baselines in Table 1 (OpenEvolve 2.634206, EvoX 2.635420; AdaEvolve 0.72031). With Luna in both roles it already beats every Terra-only baseline on Signal Processing (0.74865). The split then adds a further gain over Luna alone, which is a real finding, but the paper cannot say how much of the headline advantage would remain if the baselines were given the same cheap model. That control is cheap to run, because the baselines already support model mixing: AlphaEvolve uses a Gemini Flash and Pro ensemble; ShinkaEvolve selects models with a bandit; OpenEvolve has weighted model selection; AdaEvolve and EvoX ship in a package whose configuration supports weighted multi-model pools, and EvoX itself already uses a strategy model and a lighter implementation model. The paper never mentions this (the word "ensemble" does not appear), and Fig. 1a presents prior work as one model doing both roles. Needed: each baseline run with Luna / Flash alone, and with the same Terra+Luna / GLM+Flash mix. Without these, the paper cannot distinguish "the framework helps" from "cheap calls help".

  2. The circle-packing "new state-of-the-art" is a tolerance artifact. FrugalEvo reports 2.635996 (Table 1, Fig. 3). The best known packing for 26 circles at zero tolerance is 2.635983084919 (Packomania), and an independent September 2026 study that attacked this instance confirmed the record held at zero tolerance. The paper's own description of its solution says it applies "small radius increases within the evaluator's tolerance" (Sec. 3.2). The OpenEvolve evaluator used here allows an overlap of 1e-6 per pair; 26 radii each enlarged by about 5e-7 add 1.3e-5 to the sum, which is the gap between 2.635996 and 2.635983. ShinkaEvolve's own paper shows the same effect: its value drops below 2.635983 once the slack is removed. SwarmResearch reports exactly the same 2.635996 and CORAL 2.635985, so three systems have reached the same evaluator ceiling. At zero tolerance the packing can at best match the known value. Please report the sum of radii after shrinking the radii so that every constraint holds exactly, and remove "new state-of-the-art" from the abstract (and soften "surpassing all baselines" in the introduction) unless the exact value exceeds 2.635983084919. Table 1 lists AlphaEvolve as 2.635000; that is the paper's rounded 2.635 padded with zeros, and the precise published value is 2.63586276.

  3. The cost comparison with multi-agent systems mixes two different quantities, and the cost accounting is incomplete. The "$1.68" and "$0.55" in the abstract are the cost at which the best solution appeared (Fig. 3, "cost to best"), while the "$50" for CORAL and SwarmResearch is the total budget those runs used. The paper's own runs also spend their full budget. The two numbers are compared as if they were the same thing. Beyond that: costs are evidently computed from token counts and the App. C price list (six-decimal values), but the paper does not say so, does not report calls, input tokens, output tokens or cache hits per method, and does not say whether cached tokens were priced at the cache rate for the baselines as well (OpenAI's cache is on by default for any shared prefix over 1,024 tokens, with a 90% discount for these models). The GLM-5.3 prices in Table 7 ($1.15 / $3.50 / $0.10) do not match the public list price at the time of writing ($1.40 / $4.40 / $0.26), and the Flash prices match a 50%-off provider rate; please state the provider and date. Evaluator compute is excluded from cost (Sec. 2.1), although the discovered solutions run SLSQP, multistart hill climbing and linear programs; the introduction's "total cost" includes it. A cost-aware paper should report the token and call accounting next to the dollars.

  4. BA-AUC as reported does not match its definition, and a transform is applied to some tasks only. Eq. 4 integrates the best-so-far score over cost. For five of the ten tasks the table values cannot be that integral: on circle packing the score is about 2.636 over a $2 budget, which gives about 5.27, while Table 1 reports 1.99; on Min-Max-3, scores near 4.166 give about 8.3 and the table reports 1.97; the same holds for both Heilbronn tasks and for Cloudcast. For the other five tasks (Signal Processing, EPLB, PRISM, LLM-SQL, Transaction Scheduling) the values are consistent with the raw integral. So some tasks use a normalized score and some do not, and no normalization is described. For the two minimization tasks the highest BA-AUC is bolded as best (Min-Max-3; Cloudcast with GPT), which is fine only if a higher-is-better transform was applied; the paper does not say. Area under a best-so-far curve is the standard anytime-performance measure in optimization benchmarking (Hansen et al. 2022; López-Ibáñez & Stützle 2014; IOHprofiler), with cost in place of evaluations; the paper should cite it and present BA-AUC as that measure on a cost axis.

  5. The budget regime is far below where the baselines were designed to run, and their configurations are not stated. ShinkaEvolve with GLM-5.3 at $1 reaches a best of 2.615 on circle packing (Table 1), far below its published 2.635983; OpenEvolve with GLM reaches 2.598. Nothing in the paper says how the baselines were configured (population and island sizes, prompt length, temperature, stopping rule at the budget) or whether their defaults were adjusted for a $1 budget. The paper never runs any method above $2, so whether the ranking holds at the $20–$50 budgets used by the systems it compares against is unknown. One task at about $10 would answer this.

  6. The evidence for the ranking is thin in places, and figures and tables disagree. Three runs per cell. Several differences lie inside one standard deviation (Heilbronn Triangle with GPT: FrugalEvo 0.03480±0.0008 vs OpenEvolve 0.03417±0.0034; Cloudcast with GLM: 622.6±4.5 vs EvoX 630.0±9.2). The abstract's "matches or surpasses all baselines in final solution quality" does not hold on Heilbronn Triangle: with GLM, EvoX's mean is higher (0.03486 vs 0.03463), and with GPT the best run (0.03568) is below the baselines' 0.03653. Fig. 4 (the 10 ALE-Bench-Lite tasks, half of the benchmark) gives one average per method with no per-task numbers, no error bars and no run count. Three legend values in Fig. 5 disagree with the tables: FrugalEvo's Signal Processing mean and best (0.7587 / 0.7830 in the figure, 0.7779 / 0.7891 in Table 1) and AdaEvolve's Heilbronn Convex mean (0.0227 vs 0.0235); the other 27 agree. Fig. 1b's caption says results are averaged over three runs, while the plotted values match the "Best" column.

  7. Key settings are missing. K (number of strategies), M (refinement attempts), N0, F and ε appear in Alg. 1 and never receive values. Temperatures, seeds and context length are not stated. Cache hit rates are not reported, and the cache-efficient prompt layout, part of the second listed contribution, has no ablation (Table 3 has no row for it). Code is promised "upon publication".

  8. Novelty is modest and the framing of prior work is too strong. A strong model that plans and a cheap model that implements is the planner–executor / cascade pattern (FrugalGPT and RouteLLM are cited); AlphaEvolve's Flash/Pro ensemble was already a cost decision; EvoX already separates a strategy model from an implementation model; Borthwick 2026 evolves over a priced menu of LLMs. The statement that prior work "typically optimizes performance gain over a fixed number of iterations" is too broad: Gideoni et al. 2026 compare at a $20 API budget, EvoX reports dollar costs against ShinkaEvolve and OpenEvolve, and Xing et al. 2026 study compute allocation and argue that LLM calls are not a fair cost axis. What remains new is the specific split, the cold start, and the cache-aware prompt layout; the paper should say so and compare against the cheapest of these alternatives.

Questions for authors
  1. Please run each baseline (a) with Luna / Flash alone and (b) with the same Terra+Luna / GLM+Flash mix, using the ensemble support the baselines already have. Does FrugalEvo still win on BA-AUC and on final score?
  2. What is the circle-packing sum of radii after shrinking the radii so that all constraints hold at zero tolerance? Does it exceed 2.635983084919? Please also report the maximum pairwise overlap and boundary violation of the Fig. 3 packings.
  3. Please confirm that costs are computed from token counts, and report calls, input and output tokens, and cache hits per method and task. Were cached tokens priced at the cache rate for the baselines too? Which provider and date do the GLM prices in Table 7 come from?
  4. What exactly is q(c) in the BA-AUC numbers of Tables 1–2, task by task, and how are minimization tasks handled?
  5. What values of K, M, N0, F and ε were used, with which temperatures and seeds?
  6. How were the baselines configured (population, islands, prompt length, temperature, stopping rule), and were any defaults changed for the $0.5–$2 budgets? Can you show one task at about $10?
  7. Why do three Fig. 5 legend values differ from the tables (FrugalEvo Signal Processing mean and best; AdaEvolve Heilbronn Convex mean)? Do the tables include improvements found after the budget was exceeded, which Sec. 2.1 says the curves truncate?
  8. ALE-Bench-Lite: how many runs per task, and what are the per-task scores with standard deviations?
  9. How much of the cost saving comes from the cache-aware prompt layout? Please add that ablation and the measured hit rates.
  10. Will code, run logs and the exact baseline configurations be released before the discussion ends?
4 comments
AuthorsOpenReview ↗

Response to Reviewer xoxa (Part 1/4)

We thank Reviewer xoxa for recognizing that our work addresses the right research question with broad experimental coverage and for finding our work's ablations and qualitative analyses informative. We are also glad that the reviewer finds our presentation clear and easy to follow. Below, we address the key points raised in the review and hope our responses could resolve the concerns.


W1&Q1: The comparison does not separate the framework from the model mix, and the paper's own ablation suggests the cheap model is doing much of the work.

Thank you for raising these points. We agree that adding baselines with cheap models would strengthen our experimental comparison. Table A2, Table B2, Table C2 and Table D2 present these results on Signal Processing and Circle Packing tasks.

  • Baselines with cheap models.
  1. Table A2: Results on Signal Processing with cheaper-model-only baselines. FrugalEvo uses both the strong and cheap models (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3), while all baselines use only the cheap model (Luna or Flash). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.
MethodGPT5.6 AvgGPT5.6 BestGPT5.6 BA-AUC@2GLM5.3 AvgGLM5.3 BestGLM5.3 BA-AUC@1
FrugalEvo (Terra+Luna / GLM5.3+Flash)0.777912 ± 0.01450.7891161.481878 ± 0.03080.784035 ± 0.00340.7864130.753909 ± 0.0068
Baselines (Luna / Flash only)
OpenEvolve0.633766 ± 0.02780.6596581.168001 ± 0.03210.688770 ± 0.04890.7233740.672941 ± 0.0365
ShinkaEvolve0.556792 ± 0.09730.6690831.080353 ± 0.13770.689766 ± 0.06000.7321710.549440 ± 0.1144
AdaEvolve0.752368 ± 0.00560.7587811.470924 ± 0.01370.737765 ± 0.02240.7535940.711474 ± 0.0020
EvoX0.546776 ± 0.02370.5720481.088085 ± 0.03700.573644 ± 0.02970.5946730.569070 ± 0.0212
  1. Table B2: Results on Circle Packing with cheaper-model-only baselines. The model settings are the same as in Table A2.
MethodGPT5.6 AvgGPT5.6 BestGPT5.6 BA-AUC@2GLM5.3 AvgGLM5.3 BestGLM5.3 BA-AUC@1
FrugalEvo (Terra+Luna / GLM5.3+Flash)2.635989 ± 0.00002.6359961.992505 ± 0.00282.635422 ± 0.00082.6359900.993870 ± 0.0039
Baselines (Luna / Flash only)
OpenEvolve2.632631 ± 0.00172.6342921.990767 ± 0.00252.622509 ± 0.00792.6303670.977074 ± 0.0103
ShinkaEvolve2.635983 ± 0.00002.6359831.993307 ± 0.00162.622527 ± 0.00752.6300480.980095 ± 0.0031
AdaEvolve2.626094 ± 0.00992.6359831.987537 ± 0.00682.626615 ± 0.00452.6311050.989212 ± 0.0042
EvoX2.630705 ± 0.00032.6309721.988347 ± 0.00302.633538 ± 0.00242.6359830.982465 ± 0.0123
  • Baselines with both strong and cheap models.
  1. Table C2: Results on Signal Processing with strong+cheap-model baselines. All methods use both the strong and cheap models (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.
MethodGPT5.6 AvgGPT5.6 BestGPT5.6 BA-AUC@2GLM5.3 AvgGLM5.3 BestGLM5.3 BA-AUC@1
FrugalEvo (Terra+Luna / GLM5.3+Flash)0.777912 ± 0.01450.7891161.481878 ± 0.03080.784035 ± 0.00340.7864130.753909 ± 0.0068
Baselines (Terra+Luna / GLM5.3+Flash)
OpenEvolve0.545894 ± 0.00540.5513151.081742 ± 0.00880.715171 ± 0.00260.7177480.684850 ± 0.0157
ShinkaEvolve0.562811 ± 0.00720.5699661.078363 ± 0.00270.728831 ± 0.00450.7333070.713107 ± 0.0100
AdaEvolve0.726819 ± 0.01010.7369241.388299 ± 0.00320.760954 ± 0.00990.7708060.723474 ± 0.0008
EvoX0.547480 ± 0.02160.5691221.076352 ± 0.03390.640309 ± 0.08510.7254540.629575 ± 0.0795
AuthorsOpenReview ↗

Response to Reviewer xoxa (Part 2/4)

  1. Table D2: Results on Circle Packing with strong+cheap-model baselines. The model settings are the same as in Table C2.
MethodGPT5.6 AvgGPT5.6 BestGPT5.6 BA-AUC@2GLM5.3 AvgGLM5.3 BestGLM5.3 BA-AUC@1
FrugalEvo (Terra+Luna / GLM5.3+Flash)2.635989 ± 0.00002.6359961.992505 ± 0.00282.635422 ± 0.00082.6359900.993870 ± 0.0039
Baselines (Terra+Luna / GLM5.3+Flash)
OpenEvolve2.588702 ± 0.06692.6359831.956294 ± 0.02872.627872 ± 0.00642.6342920.973649 ± 0.0026
ShinkaEvolve2.585999 ± 0.06312.6305881.933677 ± 0.02122.599917 ± 0.03602.6359530.974708 ± 0.0106
AdaEvolve2.630226 ± 0.00022.6303811.988240 ± 0.00062.630599 ± 0.00542.6359770.984399 ± 0.0090
EvoX2.585497 ± 0.06232.6295721.921442 ± 0.00552.595936 ± 0.01582.6116890.971222 ± 0.0066

As shown in the four tables above, FrugalEvo achieves the highest average and best final scores on both tasks across different model settings. It also achieves the highest BA-AUC in most comparisons. These results demonstrate FrugalEvo’s strong final performance and overall cost efficiency.


W2&Q2: The circle-packing “new state-of-the-art” is a tolerance artifact.

Thank you for identifying this issue. We reproduced the reported packing and confirmed that its sum of radii, 2.635996082399162, relies on the evaluator’s numerical tolerance: the maximum pairwise overlap is approximately 9.99921 × 10⁻⁷, and the maximum boundary violation is approximately 4.99921 × 10⁻⁷.

To obtain a feasible packing without relying on the evaluator’s tolerance, we kept all circle centers fixed and reduced every radius by approximately 4.99961 × 10⁻⁷, with conservative downward rounding. The resulting sum of radii is 2.635983083424336, below the reference value of 2.635983084919 cited by the reviewer. Therefore, our method only outperforms the evaluated baselines, rather than establishing a new circle-packing record.

We will revise the abstract and introduction accordingly.

AuthorsOpenReview ↗

Response to Reviewer xoxa (Part 3/4)

W3&Q3&Q9: The cost comparison with multi-agent systems mixes two different quantities, and the cost accounting is incomplete.

  • Cost computation and GLM pricing (Table 7).

All costs are computed directly from the exact token counts logged during runs. We called GLM via OpenRouter with its default dynamic provider routing to ensure stable API access. Since prices across routed providers can vary slightly, we fixed a single reference price table (Table 7, based on OpenRouter's pricing prior to September 10, 2026) and used it to convert logged token counts into costs for all methods and baselines to ensure a fair comparison.

  • Tokens and calls.

We thank the reviewer for the question on token usage and cost accounting. Taking the GLM-5.3 experiments on Circle Packing as an example, Table E below reports the average number of API calls, uncached input tokens, output tokens, and cached input tokens across three runs.

  1. Table E: Token usage and cache hit rates for GLM-5.3 experiments on Circle Packing. Each run includes only complete API calls within a cumulative USD 1 budget. Cache hit rate is computed as cached input tokens divided by total input tokens.
MethodCallsUncached input tokensOutput tokensCached input tokensCache hit rate
FrugalEvo2392,531,311707,327664,70420.80%
ShinkaEvolve144544,417135,50350,9718.56%
OpenEvolve56406,561147,89529,5176.77%
EvoX54331,920170,90114,0244.05%
AdaEvolve31350,203162,38430,9568.12%

As shown in Table E, FrugalEvo achieves the highest cache hit rate (20.80%).


W4&Q4: What exactly is q(c) in the BA-AUC numbers of Tables 1–2, task by task, and how are minimization tasks handled?

We have specified task objectives in Appendices B.1 and B.2. For Signal Processing, EPLB, LLM-SQL, and Transaction Scheduling, q(c) is the best-so-far combined_score achieved within cumulative cost c. For the remaining tasks, the score follows the objective defined in the appendix.

For minimization tasks, the algorithms optimize a transformed combined_score, where higher values are better. The transformations are task-specific:

  • Min-Max-3: The objective is to minimize R = (d_max / d_min)^2. Using the reference benchmark B = 4.165849767, we define combined_score = B / R. Equivalently, R = B / combined_score.

  • Transaction Scheduling: The objective is to minimize total_cost. We define combined_score = 1 / (1 + total_cost).

Thus, maximizing combined_score is equivalent to minimizing the original objective in both tasks.

AuthorsOpenReview ↗

Response to Reviewer xoxa (Part 4/4)

W7&Q5: Key settings are missing. What values of K, M, N0, F and ε were used, with which temperatures and seeds?

We use K = 3, M = 3, N0 = 3, F = 2, and ε = 1e-7. The temperature is 0.7. We use random seeds 73, 74, and 75. We will add these to the appendix.


W5&Q6: The budget regime is far below where the baselines were designed to run, and their configurations are not stated. How were the baselines configured (population, islands, prompt length, temperature, stopping rule), and were any defaults changed for the 2 budgets? Can you show one task at about $10?

Our baselines follow the default setting in SkyDiscover [1]. Our work aims to design an algorithm that discovers new solutions while keeping costs low. As we already observe strong results at low cost, we have not extended the experiments to 10 USD. We acknowledge that larger budgets may allow the baselines to achieve better scores; however, studying performance at higher budgets is beyond the scope of this work.

[1] Liu et al. SkyDiscover. GitHub repository, 2026. https://github.com/skydiscover-ai/skydiscover.


W6&Q7: Why do three Fig. 5 legend values differ from the tables (FrugalEvo Signal Processing mean and best; AdaEvolve Heilbronn Convex mean)? Do the tables include improvements found after the budget was exceeded, which Sec. 2.1 says the curves truncate?

The discrepancies came from outdated plotting data. We have updated Fig. 5: the corrected values are 0.777912 / 0.789116 for FrugalEvo’s Signal Processing mean / best and 0.023486 for AdaEvolve’s Heilbronn Convex mean, matching the tables. Both the figure and tables only present improvements found within the budget.


W8: Novelty is modest and the framing of prior work is too strong. What remains new is the specific split, the cold start, and the cache-aware prompt layout; the paper should say so and compare against the cheapest of these alternatives.

Thanks for this comment. Our design is motivated by the characteristics of computational optimization problems, which is overlooked in prior work. We will revise the description of our novelty. Moreover, we have added results of baselines with cheap models, as shown in Table A2 and Table B2.


Q8&Q10: ALE-Bench-Lite: how many runs per task, and what are the per-task scores with standard deviations? Will code, run logs and the exact baseline configurations be released before the discussion ends?

We conducted one run per method per task and reported the average score across the ten tasks. We have included the code and run logs in the supplementary materials, where the per-task ALE-Bench-Lite scores can be found.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We thank you for your time and constructive feedback. We are encouraged that you recognize the following strengths of our work:

  • An important and practical research question. Cost-awareness in LLM-guided program evolution is a timely and important direction, and measuring progress against cost rather than iterations better reflects practical usage (xoxa, 6Cff, bU8d).
  • Broad experimental coverage. Our experiments span 20 tasks across three task families and two model families, with multiple runs per setting, and show promising gains (xoxa, 6Cff).
  • Clear presentation and reproducibility. The framework, algorithm and BA-AUC metric are clearly described (xoxa, 6Cff, bU8d), and the prompts, benchmark descriptions and price tables are provided (xoxa).

Below, we summarize the main shared concerns and how we have addressed them.

1. Separating the framework from the model mix (xoxa, 6Cff, bU8d)

All three reviewers asked whether FrugalEvo's advantage comes from its design or simply from cheaper model calls. We agree that this is the most important question, and we have added two groups of controlled baselines (cheap-model-only baselines. and model-mixing baselines.) on Signal Processing and Circle Packing, using both the GPT-5.6 and GLM-5.3 model families under the same budgets.

In both settings, FrugalEvo achieves the highest average and best final scores across both tasks and both model families. The gap is particularly clear on Signal Processing. For example, with GPT-5.6, FrugalEvo reaches an average score of 0.7779, compared with 0.7524 for the strongest cheap-model-only baseline and 0.7268 for the strongest model-mixing baseline. These results indicate that FrugalEvo's advantage comes from how it allocates the two models, rather than from model mixing or cheaper calls alone.

2. Methodological scope and novelty (xoxa, 6Cff, bU8d)

We agree that combining strong and cheap models is not new in itself. We would like to clarify that the design of FrugalEvo is motivated by the characteristics of computational optimization problems: large score gaps between solutions usually come from discovering the right optimization operator or algorithm structure, while refinement within the same approach yields smaller gains. FrugalEvo therefore assigns the strong model to strategy exploration and the cheap model to implementation and repeated refinement.

We also agree that prompt caching is a vendor-provided capability rather than an algorithmic contribution. We will present our cache-efficient prompt design as an implementation-level optimization.

3. The circle-packing result (xoxa)

We thank Reviewer xoxa for identifying this issue. We reproduced our packing and confirmed that its sum of radii (2.635996) relies on the evaluator's numerical tolerance. After shrinking all radii so that every constraint holds exactly, the sum becomes 2.635983083424, slightly below the best known value of 2.635983084919. We will therefore remove the "new state-of-the-art" claim from the abstract and revise the introduction accordingly.

4. Cost accounting and missing settings (xoxa, bU8d)

  • Cost accounting. All costs are computed from the exact token counts logged during runs. For GLM, we use a fixed reference price table from OpenRouter (prior to September 10, 2026) for all methods. We also report the number of API calls, uncached, cached and output tokens, and cache hit rates for each method. FrugalEvo achieves the highest cache hit rate (20.80% vs. at most 8.56% for the baselines).
  • Hyperparameters. We now report all key settings (K = 3, M = 3, N0 = 3, F = 2, ε = 1e-7, temperature 0.7, seeds 73–75).
  • Sensitivity analysis. As requested by Reviewer bU8d, we varied the strategy count, refinement cap and cold-start limit on Signal Processing. The default configuration achieves the highest average score, and the other settings remain competitive, showing that FrugalEvo is reasonably robust to these choices.

5. BA-AUC details and consistency (xoxa)

We clarified how the score q(c) is defined for each task and how minimization tasks are transformed into higher-is-better scores. We also corrected three legend values in Fig. 5 that came from outdated plotting data, so the figure now matches the tables. The code and run logs, including per-task ALE-Bench-Lite scores, are provided in the supplementary materials.

We thank you again for your valuable feedback, which has substantially improved our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.