- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents CoEvolve, a reflective prompt optimizer that turns failed executions into natural-language reflections and manages them as an explicit, traceable memory. These reflections are kept in a persistent store. Each candidate prompt is defined by the reflections it uses, plus a record of its parent and the operation that created it. At each step, the method runs a parent candidate on a small batch of training examples. A language model turns the parent's failures into new reflections. The parent then produces several new candidates by combining its own reflections with the new ones, up to a fixed limit. The paper defines three ways to turn reflections into prompt text. Distill condenses them into a short policy, Add appends them as separate rules, and Patch–Prune–Refresh replaces weak rules. The new candidates are scored on the same batch, so they can be compared fairly. To choose which candidate to build on next, the method rewards validation gains but penalizes candidates that use many more reflections than their parent. The main experiments use only Distill, a configuration called CoEvolve-Distill. In experiments with DeepSeek-v3.2 on FINER, Formula, HotpotQA and DDXPlus, CoEvolve-Distill achieves the highest mean score on three of the four tasks. The paper also studies selection bias, ablates the main components, and tests whether provenance helps recover from deliberately injected harmful memory.
Critical strengths
- The motivation is sound. Reflections from past failures are useful, but they become hard to track and control as they accumulate. The paper solves this by keeping every reflection in a persistent store and building each prompt from only the top-ranked ones. This keeps the prompt short without losing useful reflections.
- Growth-aware selection effectively controls prompt size.
- The ablation study covers each core mechanism.
Critical weaknesses
- The novelty over GEPA and ACE is limited. CoEvolve keeps GEPA's search loop, while its itemized memory with helpful/harmful counts closely follows ACE. The main addition is storing reflections under stable IDs, which is an incremental change.
- The gains are small and lack significance testing.
- Even with the ablation study, the benefit of each component remains unclear. For example, the largest effect of removing persistent memory lowers scores by only 0.018–0.026, comparable to run-to-run variation.
- The clarity and presentation in the methodology are very hard to follow.
Questions for authors
- Could the authors report significance test results, given that multiple runs are already available?
- When are children evaluated on the validation set?
- How do Distill siblings differ from one another?
- How is a candidate's outcome credited to individual rules?
- What exactly is the Split-B procedure?