acceptodds
Under review as a conference paper at ICLR 2027

How Can Every Rollout Count for Skill Seekers? COMPASS for Skill Proposers

Abstract

Textual-program optimization must balance proposing and evaluating candidates under a fixed rollout budget. Evaluating each candidate more extensively leaves fewer rollouts for proposing new candidates. We introduce COMPASS, which turns accumulated evaluation results into relative credit to guide subsequent proposals. On each shared instance, attaining the highest observed score earns more credit when a smaller fraction of the other candidates attain it. COMPASS averages these per-instance credits to select candidates for further revision and updates the credits as new results arrive. For revision, it assigns batches of instances using the selected candidates’ score gaps to the best score recorded among them on each instance. Each new candidate is evaluated on instances that were not used to generate it or its ancestors. Across six benchmarks, COMPASS achieves the highest average score among the compared methods for both Qwen3-8B and GPT-5.4-nano. On Qwen3-8B, it improves over GEPA by 11.8 percentage points on average and 36.7 points on AIME-2025. Ablations on Qwen3-8B show that both relative credit and matching improve the average score across the six benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive self-improvement, Foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_azp7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a method for optimizing textual prompt instructions (skills) under a limited rollout budget. The approach is inspired by genetic prompt evolution. The approach maintains a pool of candidate instructions, selecting parents using relative credit which is assigned in a tournament fashion, assigns proposal problem batches through maximum-weight matching, and evaluates revisions on separate admission batches that exclude their proposal histories. Across six benchmarks spanning diverse instruction types and domains, they report two model performance with their system, showing improvement over GEPA reported scores. They show ablations for their approach, showing benefit of relative credit and matching.

Critical strengths
  • The benchmarks report good performance improvement across a diverse set of tasks and domains.
  • The approach tackles the practical problem of a limited execution budget to evaluate new and existing submissions.
  • The paper is well written and easy to follow.
Critical weaknesses
  • The majority of improvement comes from other parts of the approach not ablated in the paper. There are ablations for relative credit and proposal matching, but it is unclear where the majority of benchmark improvement originates from in the approach.
Questions for authors
  • I believe the paper does not discuss the proposal and admission batch sizes chosen for the experiments. Is this something that can be reported?
1 comment
AuthorsOpenReview ↗

Ablation of the remaining components and the batch sizes

Thank you for the positive assessment. Your comment on the unablated parts led to the new ablations below, which the revision includes.

W1. Where the improvement originates. We ablated the remaining parts of the approach by removing each from COMPASS, alongside the credit and matching ablations (Qwen3-8B, seed 0, change in the six-task aggregate relative to COMPASS).

RemovedAdmission gatingLineage exclusionCache reuseCreditMatchingFirst three together
Δ Aggregate−2.78−2.17−1.70−2.82−1.13−4.33

Every part contributes, and the gain is shared across all five. The parts not ablated in the submission are paired admission outside the proposal history, which is the third component of our first contribution, and cache reuse (Section 2.3). Each of them carries a drop comparable in size to credit or matching. With the first three removed together, credit and matching keep the method 6.2 pp above GEPA re-run under the same settings and budget. Credit and matching together add 3.8 pp over the variant without them (three-seed mean). The revision reports this decomposition.

Q1. Batch sizes. Proposal and admission batches each contain 3 instances (bP=bA=3b_P=b_A=3). The revision reports this in Section 5.1.

Reviewer_w26bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes COMPASS for textual-program optimization under a fixed rollout budget, combining relative-credit ranking, proposal-batch matching, and paired admission that excludes proposal-history examples. The problem is practically relevant, and results across six tasks and two models suggest promise.

Critical strengths
  1. The problem is practically relevant. Improving the use of a limited rollout budget is valuable when program execution is expensive.

  2. The method explicitly separates proposal and admission evidence. Excluding proposal-history examples addresses potential bias from reusing examples.

  3. The evaluation covers six tasks and two models. The results and component ablations provide initial evidence of practical value.

Critical weaknesses
  1. The contribution over existing frameworks such as GEPA is not sufficiently established. More targeted comparisons are needed to show when and why the added mechanisms help.

  2. The theoretical analysis offers limited explanation of the core design. Equation (8) leaves ξG\xi_G uncontrolled and does not explain an advantage over mean-score ranking; routine properties could also be presented more concisely.

  3. The experiments do not fully establish stable component gains. The missing credit-by-matching ablation, lack of variability across independent runs, and unclear comparability of reused Qwen baselines limit confidence in the results.

Questions for authors

If the authors can address the following concerns with clear explanations and supporting evidence, I would consider raising my score:

  1. Clarify what Equation (8) guarantees, the limitations of leaving ξG\xi_G uncontrolled, and how its assumptions relate to the implemented evaluation procedure.

  2. Complete the credit-by-matching ablation on representative tasks and report means and variability across independent optimization runs. A comparison with greedy batch assignment would further strengthen the evidence.

  3. Provide the key hyperparameters, reflection and inference settings, and actual rollout expenditure, and establish their comparability with the main baselines.

1 comment
AuthorsOpenReview ↗

Re-run GEPA comparison, why we rank by credit and what Eq. (8) guarantees, completed credit-by-matching ablation, and settings

Thank you for the precise questions. Working through them improved the paper, and the revision includes the results and clarifications below.

W1. Contribution over GEPA. We re-ran GEPA with three seeds. It reaches 54.56±1.19 on the six-task aggregate (published 54.85), 10.4 pp below COMPASS, which has the higher mean on all six tasks. When. The gain is largest on the two mathematical reasoning tasks, AIME (+31.8 pp) and LiveBench (+18.2 pp). Why. Removing admission gating, lineage exclusion, or cache reuse from COMPASS lowers the seed-0 aggregate by 2.78, 2.17, and 1.70 pp. On AIME, where the gain is largest, admission gating and lineage exclusion have the largest effect. With these three removed together, credit and matching keep COMPASS 6.2 pp above GEPA.

W2 / Q1. Equation (8). Relative credit is a simple rebalancing. On each shared instance it rewards a candidate only when peers fail there, and gives zero when all succeed or all fail, so the ranking is driven by where candidates differ rather than by which instances each one happened to receive. Suppose A solves its one instance, which its peers also solve, and B solves one of its two instances, the one its peer fails, while both fail the other. Mean score ranks A first and relative credit ranks B first. We will add this example to Section 3.1.

Eq. (8) bounds the final pick under that ranking. The top-credit program is within a sampling term plus ξG\xi_G of the best program in the pool, where ξG\xi_G is the gap in sample mean between the top-mean and the top-credit candidate. When observations are complete, shared, and binary, the two rankings agree and ξG\xi_G is zero by Eq. (11). Otherwise ξG\xi_G stays a free, data-dependent term, and we will say so next to Eq. (8).

The scores used for this ranking must come from evaluations that did not create the candidate. In the implementation, the admission batch and its references are frozen and hidden before the child is generated, each admission batch is drawn independently, and instances in a candidate's proposal history are left out of Ωt(s)\Omega_t(s). We will mark this link next to Algorithm 1 and condense the routine properties.

W3 / Q2. Credit-by-matching ablation, variability, greedy assignment. We completed the 2×2 on all six tasks with three independent optimization runs per cell (Qwen3-8B, mean±std).

CreditMatchingHotPotQAIFBenchHoVerPUPAAIMELiveBenchAggregate
✗✗64.78±1.8440.65±1.3951.45±2.3487.42±0.5657.55±3.6765.61±1.0061.24±1.09
✓✗65.56±0.7741.90±0.9453.11±1.2690.78±1.2261.56±2.5268.25±0.6963.53±1.21
✗✓65.00±1.2041.78±1.1350.44±2.0188.44±0.9760.67±2.6767.43±1.1362.29±1.13
✓✓66.56±2.3743.31±1.1154.33±0.8893.08±0.5763.78±4.0268.97±1.8565.00±1.31

With the other component in place, matching adds 1.5 pp and credit adds 2.7 pp, and each gain is larger than without the other component. The full method has the highest mean on every task and the highest aggregate in every seed. Replacing the exact assignment with greedy assignment lowers the aggregate by 1.30 pp (seed 0) and lowers all six tasks.

Q3. Settings, expenditure, comparability. Proposal and admission batches each contain 3 instances (bP=bA=3b_P=b_A=3), the active cutoff is ρ=5\rho=5, and the wave width is w=5w=5. Qwen3-8B performs both execution and reflection with thinking enabled, temperature 0.6, top-p 0.95, top-k 20, and at most 16,384 output tokens. The rollout budget is identical across COMPASS, its ablations, and the main baselines. Actual expenditure equals this budget to within one wave on every task (HotPotQA 6,871, IFBench 3,593, HoVer 7,051, PUPA 2,426, AIME 1,839, LiveBench 1,839). The model, decoding settings, reflection prompt, minibatch size, and accounting are likewise shared. The re-run baseline reproduces its published scores, which establishes the comparability of the reused Qwen baselines.

Reviewer_gkSVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies budgeted prompt/skill optimization in the GEPA style, where a fixed rollout budget must be shared between proposing revised candidates (reflection on execution traces) and evaluating them.

Critical strengths
  1. Splitting proposal evidence from admission evidence, and reusing cached comparisons for both parent selection and batch allocation, is a natural and practically relevant budget-allocation question for reflective prompt optimization. The proposal-history set X(s,P) and the memorization construction give a crisp motivation for lineage-excluded admission.
  2. Relative credit is simple, interpretable, and update-friendly. The zero-sum per-instance formula (Prop. 2) has an obvious "unique success" intuition, handles ties and all-failure cases, and is recomputed incrementally as new results arrive. The Eq. 11 derivation clarifying that it preserves sample-mean rankings under complete shared binary data is a useful characterization of when the mechanism is innocuous.
  3. The admission protocol is methodologically sound. Freezing admission instances, references, and reference scores before generation and hiding them from the generator (§3.3, B.2) is exactly the right anti-overfitting design, and paired comparisons against a fixed reference are more efficient than independent scoring.
  4. Weakness-targeted batching via joint assignment is a good idea. Matching proposal batches to parents by cached score gaps (Eq. 3), and solving the slot–batch assignment exactly (Prop. 5), spends proposal rollouts on instances where the candidate demonstrably lags its peers.
Critical weaknesses
  1. The headline gain over GEPA is not attributable to the two named components and the full ablation already beats GEPA before either novelty is present. From Table 2: GEPA = 54.85, "w/o Credit & Matching" = 61.26, "w/o Matching" (credit only) = 62.34, COMPASS = 65.16. Thus relative credit adds only 1.08 pp and proposal matching adds 2.82 pp over the base variant, while the base variant — which still contains paired admission, lineage exclusion, reference binding, the active-set round-robin, and the score cache beats GEPA by 6.41 pp (and on AIME, the base variant reaches 57.33 vs. GEPA's 32.00, i.e., ≈90% of the total AIME gain comes from unnamed framework differences). The paper's central empirical claim ("COMPASS improves over GEPA by 10.3 pp") therefore mostly bundles a re-implementation of a GEPA-like search with a new admission/filtering pipeline. The paper needs a factorial ablation that isolates (a) admission gating with frozen references, (b) lineage exclusion, (c) the cache/reuse policy, (d) credit, and (e) matching, so the reader can see which mechanism carries the gain. As presented, attribution of the main result, the fatal-flaw check in this sub-area, is not established.
  2. Baselines are not re-run under the same harness; budget parity is asserted but the accounting is undefined. Seed, GRPO, MIPROv2, GEPA, and GEPA+Merge rows "reuse published GEPA-study task scores" (§5.2). The authors use "the rollout budget reported for GEPA" (§5.1), but the paper never defines what a rollout counts in COMPASS: parent execution on proposal batches, reflection calls, child admission execution, and frozen-reference evaluations for missing observations (B.2: "Missing reference observations are evaluated before child generation and stored") all consume executions; whether each is charged against B is not stated. Reflection prompts, generator temperatures, model versions, and stopping rules can also differ from the published GEPA runs. GEPA should be re-run inside the COMPASS harness with the same generator, budget accounting, and seeds, at least for the six Qwen tasks; otherwise the 6.41 pp gap between the un-ablated framework and GEPA is plausibly an implementation artifact.
  3. No repeated runs, no variance, no significance testing.
Questions for authors
  1. Please re-run GEPA (and ideally GEPA+Merge) inside the COMPASS harness with the same generator, prompts, temperatures, and rollout accounting. What fraction of the 10.3 pp aggregate / 31.3 pp AIME gap survives?
  2. Can you provide ≥3 seeds per task–method with paired confidence intervals and significance tests for Tables 2–3?
1 comment
AuthorsOpenReview ↗

GEPA re-run, three-seed results with significance tests, and ablation of all five mechanisms

Thank you for the careful review. The experiments you asked for are complete, and they sharpen what the paper can say about where its gain comes from.

W2 / Q1. GEPA re-run. We re-ran GEPA inside the COMPASS harness with the same model, reflection prompt, decoding settings, stopping rule, and rollout accounting, on all six tasks with three seeds.

Qwen3-8B (mean±std, 3 seeds)HotPotQAIFBenchHoVerPUPAAIMELiveBenchAggregate
GEPA (re-run)61.55±2.6738.83±1.4751.89±2.2292.31±0.6032.00±4.1650.81±1.8554.56±1.19
w/o Credit & Matching64.78±1.8440.65±1.3951.45±2.3487.42±0.5657.55±3.6765.61±1.0061.24±1.09
w/o Matching65.56±0.7741.90±0.9453.11±1.2690.78±1.2261.56±2.5268.25±0.6963.53±1.21
w/o Credit65.00±1.2041.78±1.1350.44±2.0188.44±0.9760.67±2.6767.43±1.1362.29±1.13
COMPASS66.56±2.3743.31±1.1154.33±0.8893.08±0.5763.78±4.0268.97±1.8565.00±1.31

The re-run reproduces the published scores (published aggregate 54.85, AIME 32.00), and the entire gap survives, with 10.4 pp on the aggregate and 31.8 pp on AIME. Re-run GEPA+Merge (seed 0) is 11.0 pp below COMPASS.

Rollout accounting. A rollout is one execution of a program on one instance. Parent executions on proposal batches, child executions on admission batches, and reference executions for missing observations are charged against BB. Cached observations are reused without charge. Reflection calls follow GEPA's accounting, which budgets program executions. COMPASS, its ablations, and the re-run baselines spend the same BB per task. The revised Section 5.1 states this definition.

W3 / Q2. Seeds, confidence intervals, significance. The table above reports three optimization seeds for every task and method of Table 2, and the table below does the same for Table 3.

GPT-5.4-nano (mean±std, 3 seeds)HotPotQAIFBenchHoVerPUPAAIMELiveBenchAggregate
MIPROv262.22±0.6947.51±1.3052.67±1.5375.60±1.0330.00±3.3345.08±1.2952.18±1.19
GEPA57.11±1.6848.53±1.3762.11±0.8480.42±0.6231.33±5.8134.93±1.7152.41±1.40
COMPASS59.11±1.0251.25±1.5360.22±1.1784.16±1.3642.22±3.0144.69±0.4356.94±1.35

Runs of different methods are independent, so we apply Welch's t-test to the six-task aggregate. On Qwen3-8B, the gain of COMPASS over GEPA has 95% CI [7.6, 13.3] pp (p < 0.001), and its gain over the variant without credit and matching has 95% CI [1.0, 6.5] pp (p = 0.020). On GPT-5.4-nano, the gain has 95% CI [1.4, 7.7] pp over GEPA (p = 0.016) and [1.9, 7.7] pp over MIPROv2 (p = 0.011).

W1. Attribution of the gain. We ablated each mechanism the reviewer lists by removing it from COMPASS, and also removed (a), (b), and (c) together (Qwen3-8B, seed 0, change relative to COMPASS).

Removed(a) admission gating(b) lineage exclusion(c) cache reuse(d) credit(e) matching(a), (b), (c) together
Δ Aggregate−2.78−2.17−1.70−2.82−1.13−4.33

Every mechanism contributes, and the gain is shared across all five. The variant without credit and matching retains (a), (b), and (c), and each of these carries a drop comparable in size to credit or matching. On AIME, (a) and (b) have the largest effect. With (a), (b), and (c) removed together, credit and matching keep the method 6.2 pp above re-run GEPA on the aggregate and 24.7 pp above it on AIME. Credit and matching therefore account for a large share of the gain on their own. All five mechanisms belong to the proposed method. Paired admission outside the proposal history is the third component of our first contribution, and cache reuse is described in Section 2.3. Credit and matching are fully crossed with three seeds in the table above and add 3.8 pp over the variant without them. The revised abstract and Section 5 report this decomposition, with the re-run GEPA rows in Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.