- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies LLM-driven evolutionary discovery (AlphaEvolve-style) under a fixed dollar budget. It argues that existing adaptive controllers (AdaEvolve, EvoX) assign credit from score progress alone, ignoring that LLM actions have different realized costs. The proposed controller, CostAda, divides a progress signal by a log-scaled cost factor. It also conditions on the remaining budget: credit shifts from local to global improvement, and the cost penalty strengthens as budget runs out. This credit drives three controls: local exploration intensity, UCB-based frontier allocation, and budget-gated "guide" interventions. A worst-case separation result (Prop. 1) shows cost-blind controllers can lose most attainable value as frontier count and cost ratio grow. Experiments use 8 benchmarks, two backbones (GLM-5 at $1, GPT-5.4 at $5), and 3 seeds. CostAda reports the best mean final quality on all 16 benchmark–backbone pairs, and reaches the strongest baseline's final quality with at most half the budget on 13 of 16.
Critical strengths
- The problem is practical and timely. Dollar-denominated budgets matter for these systems, and comparing methods at budget cutoffs rather than iteration counts is the right protocol (Fig. 4 shows the same budget yields 53, 60, and 80 iterations across methods).
- Reporting is thorough: per-cutoff tables, normalized AUC, budget overshoot accounting, a controller trace, fixed constants across benchmarks, and released code.
- Gains are consistent in direction across both backbones. The independent B=0.5 runs (Table 9) are a good check that the method adapts to the specified budget rather than benefiting from truncation.
Critical weaknesses
- Theory and method are loosely connected. Prop. 1's cost-aware policy ranks by gain per cost, but CostAda divides by 1 + λᶜ·log(1 + c̃), a much weaker correction. For equal gains, a 6× costlier action gets only ~21% less credit early in the run and ~1.7× less at budget exhaustion, not 6×. The separation also needs large K, while all reported runs use K = 2 (App. C.3). The result itself is close to known bandits-with-knapsacks arguments (App. B).
- Cost is not shown to be an informative signal. All modes use the same context structure (App. C.2), so cost may not differ systematically by frontier; the 4–6× variation (Fig. 4) could be output-length or retry noise. No analysis links cost to frontier, mode, or later gain. The gain is also not "more iterations per dollar": EvoX runs more iterations (80 vs. 60, Fig. 4) yet scores lower. Sec. 5.5 credits the largest gain to the new low-yield guide trigger, so intervention timing, not cost-calibrated credit, may drive part of the improvement.
- Ablations raise consistency questions.
- Ablated variants land both far below and well above AdaEvolve (Heilbronn Triangle, GLM-5: 0.0228 / 0.0251 vs. 0.0303; Signal Processing, GLM-5: 0.6851 vs. 0.6393). How does CostAda's scaffold differ from AdaEvolve's?
- CostAda's ablation-table numbers differ from the main tables except on Signal Processing (e.g., 2.6281 vs. 2.6306; AUC 0.9517 vs. 0.9295), suggesting separate runs.
- Only 3 of 8 benchmarks are ablated.
- Missing baselines. The obvious one is AdaEvolve with reward divided by realized cost. The cited budget-aware discovery methods (Xing et al., 2026; Yang et al., 2026b) are not compared, and cost-aware Bayesian optimization (e.g., EI per unit cost) is not discussed.
- Weak statistical support. With 3 seeds and no significance tests, several "wins" fall within one std (e.g., Third Autocorrelation GLM-5: 1.4658 ± 0.0037 vs. 1.4685 ± 0.0014; Circle Packing GPT-5.4: 2.6248 ± 0.0046 vs. 2.6196 ± 0.0030). Some half-budget wins are razor-thin (2.6211 vs. 2.6196). Under GPT-5.4, three tasks account for ~75% of the aggregate margin over AdaEvolve.
- Circle Packing reference. Cited work (ShinkaEvolve, TTT-Discover) already reports ~2.635983 for n=26. CostAda's 2.6360 matches this rather than setting a new best. Please report more digits and state whether feasibility is checked exactly or with slack.
- Scale and sensitivity. Runs are short (~40–80 iterations). About ten controller constants (Table 4) have no sensitivity analysis or stated selection procedure. Evaluator cost is excluded (App. A).
Questions for authors
- How much of realized-cost variance is explained by frontier or sampling mode versus per-call noise? Is cost predictive of subsequent gain?
- Can you report AdaEvolve with plain gain-per-cost credit, and a CostAda variant with all budget-aware components removed? How does the latter compare to AdaEvolve?
- Why do the CostAda numbers differ between the ablation and main tables for some benchmarks, and how were runs selected?
- How does CostAda behave with more frontiers and longer runs (hundreds of iterations)?
- Can you add more seeds or significance tests for the main claims?
- How were the controller constants chosen, and how sensitive are results to them?
Thank you for the additional experiments and clarifications. The new comparisons address several of my experimental concerns and make the results more convincing. I have raised my soundness score to 3, but will keep my overall rating unchanged, as I still have reservations about the methodological novelty.