- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes component-aware feedback for LLM-guided program evolution. It records which functions and constants changed, along with the corresponding changes in evaluation metrics, and uses this history to guide future edits. Comparisons with the parent program guide individual mutations, while comparisons with the initial seed help suggest new directions when search stalls. The authors integrate the method into AdaEvolve and evaluate it mainly on retrieval and reranking across twelve BRIGHT datasets, reporting better held-out ranking quality and reaching comparable training scores in fewer iterations. Under a cost-aware objective, it also finds pipelines that use fewer tokens per query than AdaEvolve’s.
Critical strengths
-
I think the idea is clear and well motivated. I believe the idea of showing the LLM which components changed and how the metrics responded is novel and helpful in the evolution.
-
The paper is cleanly written, readable, and the results are encouraging, the authors show better average held-out ranking quality than across twelve BRIGHT datasets, and they use AdaEvolve as the backbone and the reference. I believe the cost-aware experiments are also valuable.
Critical weaknesses
-
The results show an improvement over AdaEvolve, but the advantage over other feedback-based methods is a bit unclear. AdaEvolve is a meaningful baseline for testing the proposed addition. Still, a comparison with at least one related method, such as ReEvo, DeltaEvolve, or GEPA’s optimize_anything extension, under comparable conditions would help establish the benefit more clearly. I would also like to understand what the component-aware representation itself adds. Table 4 suggests that richer metrics account for about half the gain on the three evaluated datasets, with attribution providing the rest. This is useful evidence, but it does not distinguish the benefit of organizing changes by functions and constants from simply showing previous edits alongside their metric changes.
-
This is less critical, but I think the authors need to tone down their claim about search-compute savings. Section 3.2 suggests that one-third of the iterations means roughly one-third of the compute, but evaluation costs can change as programs evolve. For example, the final quality-only pipelines use 71.8k tokens per query versus AdaEvolve’s 53.7k. These figures do not establish total search cost, but they show why fewer iterations need not mean proportional savings.Otherwise, the claim could be limited to fewer search iterations.
Questions for authors
-
I noticed that in the tables, several per-dataset gains are small relative to the reported variation. Do you think using three-seeds is enough for these settings?
-
In Prompt 7, the current best score is first listed as 0.607106, but later in the analysis framework part, it is given as 0.380053, which matches the seed score. Was this also present in the experimental prompts, or is it just an appendix error?
-
Could you clarify what exactly changes in the “without attribution” ablation? Does the model still see the same recent attempts and their metric changes?
-
Are the same query splits used across methods and runs, and do they stay fixed during each search? I also noticed that the seed program’s training score varies across runs, could you explain what causes this since you're also using greedy reranking?
-
As an optional ablation, have you tried local and global feedback separately? I’m curious whether global feedback really helps the search recover when it stalls.
3 comments
Thank you for the additional experiments. The comparison with plain diffs and LLM-written summaries, and the token measurements address my points. I’ve increased my score from 3 to 4.
Thank you for raising your score. We are glad our additional experiments and revisions have addressed your concerns. Thank you again for your time and constructive feedback.
Response to Reviewer 6EZr
Thank you for taking the time to review our paper and for your thoughtful comments.
What the component-aware representation adds (Weakness 1)
The component-level representation itself drives the gain, as describing edits by their components outperforms both plain code diffs and LLM-written deltas under the same metrics. We extended the ablation to six BRIGHT datasets and added two other ways of describing past edits, plain code diffs and LLM-written deltas following DeltaEvolve (Table B, paper Table 4). Adding the metrics alone barely changes the score, so most of the gain comes from the attribution. It gives the best mean and reaches AdaEvolve’s best score fastest. Showing previous edits alongside their metric changes is therefore not enough on its own, and how each edit is described matters.
Search compute claim (Weakness 2)
We now report compute directly instead of inferring it from iterations. Our search reaches AdaEvolve’s best score with half the mutation and evaluation tokens AdaEvolve needs to get there (Table A), and 77% and 55% under the cost-aware objective (paper Table 2). We have revised Section 3.2 accordingly (paper Tables 1, 2 and 7). Every iteration pairs one mutation call with one evaluation, and for many tasks, such as training recipes or large scale distributed systems, the evaluation dominates. Reranking is one such case, where evaluation takes over 97% of all tokens the search spends (Table A). Each iteration saved is therefore one fewer expensive evaluation. Our mutation prompts are longer, but mutation is such a small share of all tokens that saved iterations outweigh them.
Are three runs enough? (Question 1)
Three runs per dataset is the typical configuration, as used in the original AdaEvolve paper. We evaluate our method by its mean across the 12 datasets rather than by any single dataset or run. Our method achieves a higher mean test score on 9 of 12 datasets, and in terms of raw best score, we find a better solution on 10 of 12 datasets (paper Table 1). The gains hold in the same direction across every setting we test, the main results, ablation, OpenEvolve, transfer, and a different mutator model (Tables B to D, paper Table 3).
Prompt 7 score (Question 2)
This is an error in the appendix only. We have corrected it in the revision.
What the "without attribution" ablation changes (Question 3)
The without attribution ablation adds the decomposed metrics as feedback and nothing else. It does see a basic version of recent attempts with just the score changes, which the default search controller already supports. Our method replaces this basic version of recent attempts with the score changes plus the component-aware feedback. We have clarified this in Section 3.2 (Ablations).
Query splits and seed score variation (Question 4)
Yes, every method and run uses the same fixed 60/40 train/test split per dataset, and it does not change during a search. The seed's training score varies because vLLM batching is nondeterministic, which changes outputs slightly even under greedy decoding (vLLM batch invariance docs). This affects every method equally.
Local versus global feedback (Question 5)
We have not tried local and global feedback separately, and leave this ablation to future work.
We hope the additional experiments address your concerns and would appreciate your reassessment in light of these results. Thank you again for your time.