- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper adopts a division of labor between a stronger, more expensive LLM for strategy exploration and a cheaper LLM for code implementation, aiming to improve performance gain per unit cost. It further optimizes prompt prefixes to enable cache-efficient generation. Experiments on 20 tasks show that FrugalEvo achieves comparable or better performance than four existing baselines while improving Budget-Aware AUC (BA-AUC), which measures the quality of the best-so-far solution over cumulative LLM expenditure and thus evaluates optimization performance under cost constraints.
Critical strengths
-
This paper uses a relatively simple division of labor between LLMs to reduce inference costs. This is a timely and practically relevant direction, as the high cost of LLM usage remains a significant barrier for researchers with limited computational budgets.
-
The framework and pipeline are clearly described with an algorithm box, a clean high-level schematic, and a precise definition of BA-AUC. The rationale behind each component (strategy explorer, solution generator, search memory) is easy to follow.
Critical weaknesses
-
Limited methodological novelty and overextended claims regarding "cost-awareness" and caching. (1) The "cost-aware" logic lacks dynamic adaptation—it merely enforces a static, predefined division of labor (strong LLM for strategy exploration, cheap LLM for code implementation) without any dynamic, online cost-performance feedback or adaptive routing. (2) Prompt caching is an existing infrastructure capability provided by commercial API vendors rather than an algorithmic contribution of this paper; the authors merely apply standard prompt engineering by ordering static instructions first to exploit vendor prefix-matching rules. (3) The reliance on vendor-specific pricing and caching discount schemes raises significant generalizability concerns, as the reported cost advantages depend heavily on current commercial API pricing models and may diminish as base inference costs continue to drop.
-
Confounded Baseline Evaluation and Missing Sensitivity Analysis. (1) Unfair Baseline Comparisons: Baselines are restricted to a single high-cost model, whereas FrugalEvo uniquely benefits from a dual-model cascade and prompt caching. Equipping baselines with a similar dual-model setup is necessary to isolate true algorithmic search gains from API price arbitrage. (2) Missing Hyperparameter Sensitivity Analysis: The paper lacks sensitivity analysis for crucial search hyperparameters—including strategy count , refinement cap , and cold-start limit —leaving framework robustness and hyperparameter tuning across domains unverified.
Questions for authors
-
How much of FrugalEvo's improvement in BA-AUC and optimization performance can be attributed separately to the evolution strategy, heterogeneous model collaboration, and cache-efficient prompt construction? In particular, could the authors provide a caching-on versus caching-off comparison under otherwise identical settings?
-
Could representative baselines, such as OpenEvolve or EvoX, be evaluated with the same two-model cascade and comparable prompt-caching support under matched cost budgets? This comparison would help determine whether FrugalEvo's advantage comes from its evolution strategy or primarily from access to heterogeneous models and lower-cost inference. Results from such a controlled comparison could materially affect my assessment of the paper's central contribution.
-
What makes the proposed framework cost-aware at the search level? Beyond evaluating methods using BA-AUC under a fixed cost budget, does FrugalEvo adapt model-call allocation, strategy exploration, or refinement decisions in response to observed cost-performance feedback? If so, could the authors clarify the mechanism and provide evidence of its effect? If the allocation is largely predefined, could the authors clarify how the claimed contribution differs from a fixed two-model cascade combined with cache-efficient prompting? This clarification would help establish the methodological scope and novelty of the work.
4 comments
Response to Reviewer bU8d (Part 2/2)
(2) Sensitivity Analysis. We agree that adding sensitivity analysis is important. We have conducted a sensitivity analysis of strategy count , refinement cap , and cold-start limit on the Signal Processing task using GPT-5.6. The results are presented in Table E.
- Table E: Sensitivity analysis of strategy count (K), refinement cap (M), and cold-start limit (C) on Signal Processing using GPT-5.6 under a USD 2 budget per run. We report mean ± standard deviation and best scores across 3 runs.
| Configuration | Avg ± Std | Best |
|---|---|---|
| FrugalEvo (K=3, M=3, C=3) | 0.777912 ± 0.0145 | 0.789116 |
| K=1, M=3, C=3 | 0.734899 ± 0.0263 | 0.761151 |
| K=5, M=3, C=3 | 0.758914 ± 0.0204 | 0.779275 |
| K=3, M=1, C=3 | 0.774967 ± 0.0083 | 0.783267 |
| K=3, M=5, C=3 | 0.746496 ± 0.0246 | 0.771093 |
| K=3, M=3, C=1 | 0.718922 ± 0.0149 | 0.733778 |
| K=3, M=3, C=5 | 0.776601 ± 0.0314 | 0.807969 |
As shown in Table E, the default configuration achieves the highest average score among the evaluated settings. Increasing C to 5 yields a similar average and the highest best score, but also greater variability across different runs. Moreover, other parameter changes lead to lower average scores.
Q1: How much of FrugalEvo's improvement in BA-AUC and optimization performance can be attributed separately to the evolution strategy, heterogeneous model collaboration, and cache-efficient prompt construction? Could the authors provide a caching-on versus caching-off comparison under otherwise identical settings?
Thank you for raising this question.
- Performance improvement attribution.
The performance improvement of FrugalEvo is most likely attributed to our model collaboration design for the computational optimization tasks. As discussed in our response to W1, this design is motivated by our observations about the characteristics of these optimization tasks.
- Caching-on and caching-off comparison.
Our cache-efficient prompt construction is intended primarily to reduce API costs. We analyzed Circle Packing runs of FrugalEvo using GLM 5.3 and its Flash variant across three runs, each with a 1 USD budget.
To isolate the caching discount, we kept the recorded prompts, model calls, generated programs, and evaluation results identical. For caching-off, we recalculated costs by charging all input tokens at the regular input rate, while keeping output pricing unchanged.
The average cost across three runs is 0.776558 USD with caching enabled and 0.837117 USD with caching disabled. Caching reduces the average cost by 0.060559 USD, corresponding to an aggregate cost reduction of 7.23%.
Q3: What makes the proposed framework cost-aware at the search level?
Thanks for this question. FrugalEvo is designed to improve solution quality under a fixed cost budget; it does not dynamically allocate model calls based on observed cost–performance feedback.
The assignment of strategy exploration to a stronger model and implementation to a cheaper model is predefined, motivated by our observations about the characteristics of computational optimization tasks.
Thank you for the additional experiments and clarifications. I appreciate the efforts to introduce model-mixing baselines and conduct hyperparameter sensitivity analysis. These additions improve the experimental evidence and partially address my concerns regarding baseline fairness and robustness.
However, my primary concern regarding methodological novelty remains unresolved. As the authors explicitly acknowledge, FrugalEvo does not dynamically adapt model allocation, strategy exploration, or refinement decisions based on observed cost-performance feedback. Instead, it relies on a predefined division of labor between a stronger model for strategy exploration and a cheaper model for code implementation. While this is a reasonable and practically useful design choice, the underlying idea is relatively straightforward and, in my view, represents more of a lightweight engineering heuristic than a substantial methodological contribution.
Overall, the additional experiments provide stronger empirical support for the practical effectiveness of FrugalEvo, but they do not fundamentally change my assessment of its methodological contribution. I acknowledge its practical value; however, I remain unconvinced that the proposed framework offers sufficient methodological novelty for acceptance at ICLR.
Therefore, I maintain my original rating of 2 (Weak Reject).
Thank you for acknowledging our additional experiments and for the thoughtful follow-up. We agree that FrugalEvo uses a fixed division of labor rather than dynamically adapting model allocation, and we would like to briefly clarify why we view this design as more than an engineering heuristic.
First, the division of labor in FrugalEvo is derived from an empirical insight into computational optimization problems: large performance gaps between solutions mainly stem from whether the right operator or algorithm structure is discovered, while refinement within the same operator yields only marginal gains. FrugalEvo is built directly on this insight, allocating the strong model to strategy exploration and the cheap model to repeated refinement. We believe that identifying where model capability matters most in the evolutionary search process for computational optimization tasks is itself a methodological contribution.
Second, the model-mixing baselines provide evidence that the gains come from this specific design rather than from simply combining models. These baselines use exactly the same models and budget as FrugalEvo, yet still underperform it (Tables C and D). This indicates that how capability is allocated across the search process, rather than model mixing per se, is the key to FrugalEvo’s effectiveness.
Finally, we view the simplicity of FrugalEvo as a strength: it can be easily integrated into existing evolutionary frameworks. We agree that dynamic model allocation is a promising direction, and we will discuss it as future work in the revised version.
We sincerely appreciate your time and constructive feedback throughout the discussion!
Response to Reviewer bU8d (Part 1/2)
We thank Reviewer bU8d for finding our work explores a timely and practically relevant direction and for appreciating our framework and pipeline are clearly described. We are also glad that the reviewer finds the rationale behind each component in our work is easy to follow. Below, we address the key questions raised and hope our responses resolve the concerns.
W1: Limited methodological novelty and overextended claims regarding "cost-awareness" and caching.
Thanks for pointing out these concerns. We address them separately below.
(1) The “cost-aware” logic lacks dynamic adaptation. We agree that our method does not perform adaptive model routing. The assignment of strategy exploration to a stronger, more expensive model and code implementation to a cheaper model is fixed. This design is motivated by the observation that, in computational optimization tasks, algorithmic structure and the choice of optimization operators can substantially affect solution quality. We therefore allocate the stronger model to explore the design strategies, while using the cheaper model to implement and refine them. Our intended contribution is this strategy–implementation decomposition and its empirical effectiveness under a fixed budget. We will revise the description to make this scope explicit.
(2) Role of prompt caching. We agree that prompt caching is a vendor-provided capability, rather than an algorithmic contribution of our work. Our discussion was intended to highlight the practical importance of cache reuse in evolutionary search and how prompt organization can exploit this capability to improve cost efficiency. In the revision, we will clarify that this is an implementation-level optimization.
(3) Pricing dependence and generalizability. We agree that the magnitude of cost savings depends on model prices and caching discounts. Our primary focus is on how to combine two models with different costs and capabilities to obtain better solutions under a fixed cost budget. We will explicitly acknowledge this as a limitation in the revision.
Overall, our work aims to discuss how evolutionary search can be designed to maximize performance gains per unit cost rather than per iteration. This perspective has received limited attention in prior work, yet it is important in practice.
W2&Q2: Confounded Baseline Evaluation and Missing Sensitivity Analysis.
Thanks for this comment. We address each concern below.
(1) Unfair Baseline Comparisons. We agree that adding baselines with a similar dual-model setup is necessary. Table C and Table D present results of model-mixing baselines.
Table C: Results on Signal Processing with model-mixing baselines. Both FrugalEvo and all baselines use the expensive and cheap models together (Terra+Luna for GPT5.6, GLM5.3+Flash for GLM5.3). The budget is 2 USD per run for GPT5.6 experiments and 1 USD per run for GLM5.3 experiments.
Table D: Results on Circle Packing with model-mixing baselines. The model settings and budgets are the same as in Table C.
As shown in Table C and Table D, FrugalEvo surpasses all the baselines on both tasks given the same budget. This indicates that our model collaboration design is more efficient on these mathematical optimization tasks.