- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CoCoEvolve, an evolutionary optimizer combining a strategy ledger, on-demand retrieval of search history, and an agentic mutator that investigates and tests candidate changes. Evaluations span data agents, SQL pipelines, data engineering, and prompt, mathematical, and systems optimization.
Critical strengths
- The evaluation is unusually broad and includes substantial held-out improvements on practically relevant workloads.
- Mechanism ablations and detailed execution traces provide useful evidence about how historical retrieval and pre-commit testing support structural improvements beyond prompt editing.
Critical weaknesses
- Search variability substantially weakens the component-level conclusions. Appendix E.3 reports 114/150 test successes for a second full DABStep run, versus 137/150 for the flagship. Most primary ablations use one search and repeatedly reuse the flagship. Moreover, ledger-only underperforms parent-only on DABStep.
- Baseline comparability is incomplete. Appendix G.1 restricts GEPA to instruction edits while CoCoEvolve can modify stored procedures and dynamic tables. Equal iteration counts also do not establish equal token, probing, or evaluation costs.
- Appendix I.1 states that adherence tests were not used uniformly in data-engineering runs. This leaves a potentially consequential experimental condition insufficiently controlled or documented.
Questions for authors
- Do the main improvements and ablation rankings persist across independent searches under matched total computational budgets?
- Can the authors provide comparisons with equally expressive editable artifacts and specify adherence-test usage for every data-engineering result?
2 comments
I thank the authors for their detailed responses, which have largely addressed my concerns. I remain positive about the paper and maintain my original score.
We thank the reviewer for the thoughtful feedback. We address the questions and concerns below:
Q1 / W1 / W2
Please refer to the general response.
Q2 / W2 / W3: Comparability of Editable Artifacts and Adherence-Test Usage
mh_tools.py; 115/150 and 83/150 vanilla) and CoCoEvolve - as well as CoCoEvolve's internal ablation arms A1 (static one-shot mutator, 47/150 test), A2 (interactive agentic mutator without pre-committest_draft, 98/150 test), the history ablations, and C1--C2 (Appendices E.2--E.4) - have unrestricted access to edit the entire specification, including instructions, tool schemas, stored procedures, and dynamic tables. While external GEPA is by design a reflective prompt/instruction optimizer, our internal A1/A2, the no-meta-reflection arm, and C1 controls and Meta-Harness provide direct comparisons on the exact same full-specification editing surface.pytest test_kg.pyduring a given task attempt), not to any difference in supplied inputs or harness permissions across methods. No method received assistance unavailable to the others.