- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed a method for optimizing textual prompt instructions (skills) under a limited rollout budget. The approach is inspired by genetic prompt evolution. The approach maintains a pool of candidate instructions, selecting parents using relative credit which is assigned in a tournament fashion, assigns proposal problem batches through maximum-weight matching, and evaluates revisions on separate admission batches that exclude their proposal histories. Across six benchmarks spanning diverse instruction types and domains, they report two model performance with their system, showing improvement over GEPA reported scores. They show ablations for their approach, showing benefit of relative credit and matching.
Critical strengths
- The benchmarks report good performance improvement across a diverse set of tasks and domains.
- The approach tackles the practical problem of a limited execution budget to evaluate new and existing submissions.
- The paper is well written and easy to follow.
Critical weaknesses
- The majority of improvement comes from other parts of the approach not ablated in the paper. There are ablations for relative credit and proposal matching, but it is unclear where the majority of benchmark improvement originates from in the approach.
Questions for authors
- I believe the paper does not discuss the proposal and admission batch sizes chosen for the experiments. Is this something that can be reported?
Ablation of the remaining components and the batch sizes
Thank you for the positive assessment. Your comment on the unablated parts led to the new ablations below, which the revision includes.
W1. Where the improvement originates. We ablated the remaining parts of the approach by removing each from COMPASS, alongside the credit and matching ablations (Qwen3-8B, seed 0, change in the six-task aggregate relative to COMPASS).
Every part contributes, and the gain is shared across all five. The parts not ablated in the submission are paired admission outside the proposal history, which is the third component of our first contribution, and cache reuse (Section 2.3). Each of them carries a drop comparable in size to credit or matching. With the first three removed together, credit and matching keep the method 6.2 pp above GEPA re-run under the same settings and budget. Credit and matching together add 3.8 pp over the variant without them (three-seed mean). The revision reports this decomposition.
Q1. Batch sizes. Proposal and admission batches each contain 3 instances (bP=bA=3). The revision reports this in Section 5.1.