- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies program optimization for a fixed mixture of heterogeneous requests without task labels at inference. Adaptive-GEPA jointly edits a natural-language router, specialist descriptions, and executable specialist programs, and uses validation-request overlap to align and inherit description-program pairs across search branches. On HotpotQA, AIME, PUPA, and LiveBench-IF, the authors report family-mean test scores of 70.6 versus 62.5 for single-program GEPA-FPA-ALL on Qwen3-8B, and 71.5 versus 69.8 on GPT-4.1. The search records show changes in task partitioning and cross-branch program reuse.
Critical strengths
- The explicit router-description-program representation and footprint-based alignment address the slot-mismatch problem when recombining multi-specialist branches. Figure 7 demonstrates cross-slot alignment, and Appendix H records 10 accepted merges that inherited active specialist pairs unique to both parents.
- The full system improves the four-family mean over GEPA-FPA-ALL, a single-program baseline that can also learn internal dispatch and multi-stage workflows (Tables 2-3). Adaptive-GEPA scores lower on Qwen LB-IF (85.0 versus 91.4), but reaveraging the same selected candidates without PUPA still leaves advantages of 2.81 points on Qwen and 2.91 on GPT-4.1 (Appendix G.1).
Critical weaknesses
- Unclear handling of corrupted archive entries in the GPT-4.1 run. Appendix L (lines 1301-1306) reports zero PUPA vectors for candidates 15-22 during an evaluator outage and a reset merge counter after resumption. The final candidate was generated after recovery, but the paper does not say whether affected entries were repaired before later parent selection or give the actual merge count relative to the original cap. With a reported GPT-4.1 margin of 1.62 points (Table 8), I would like to know how much subsequent search depended on those entries and the extra merges.
- Unclear development decisions after viewing test curves. Appendix L (lines 1296-1299) states that a callback evaluated every fifth accepted candidate and that developers could see the curves, although the optimizer selected candidates using validation scores alone. The paper does not state whether viewing these curves led to changes in the algorithm, hyperparameters, or protocol, or whether any final evaluation used a test set that had not informed those decisions.
Questions for authors
Key Questions For Authors
- In the GPT-4.1 experiment, were outage-affected entries 15-22 repaired or reevaluated before they could be sampled as parents? What was the actual total merge count compared with the intended cap (Appendix L)?
- Were any components, prompts, hyperparameters, or protocol settings changed after inspecting the test callback curves? Did any final evaluation use a test set that had not informed those development decisions (Appendix L)?
- Why does GEPA-FPA-ALL use a single global execution profile while Adaptive-GEPA can select profiles per specialist (Section 4.1; Appendix D, lines 767-771)? How much of the gap remains if both methods have the same profile flexibility?
- Appendix I.2 projects the aligned children's scores, whereas Table 12 includes a measured historical index child. Could you execute both aligned and index-based children on a few identical parent pairs from the main run, using the same scoring and capacity rules?
- PUPA accounts for about 74% of the Qwen difference in the four-family mean (Appendix G.1), but the saved test results lack per-request score vectors. Could you reevaluate the frozen final candidates on PUPA and separately report answer quality, privacy leakage, and empty delegation-request rates?
Additional Comments
- Could you briefly explain why GPT-4.1 was chosen as the second actor model?
- Please specify the exact merge scheduling rule. Appendix A (lines 649-651) describes the five-example check; how are those examples divided between cases favoring each parent and ties?
- GEPA-FPA-ALL's HotpotQA score is 54.0 in Figure 1(b) and 54.1 in Table 2. Please reconcile the two values.
8 comments
Thanks for your review (2/n)
Q3: Why GEPA-FPA-ALL Uses One Global Profile and Same-Profile Evaluation
"Why does GEPA-FPA-ALL use a single global execution profile while Adaptive-GEPA can select profiles per specialist (Section 4.1; Appendix D, lines 767-771)? How much of the gap remains if both methods have the same profile flexibility?"
GEPA-FPA-ALL uses one global profile because an execution profile (fast or deep) is specified once per program and it optimizes a single monolithic program—and locking all four active Adaptive-GEPA specialists to GEPA-FPA-ALL's deep profile preserves our full gap (70.76 vs. 62.47, +8.29 points).
Experiment setup:
- To isolate the effect of our learned specialist structure from per-specialist
fast/deepprofile choices, we set all four active specialists in our selected Qwen3-8BAdaptive-GEPAlibrary todeep—matchingGEPA-FPA-ALL(as well as the initial program, which usesdeepacross every task). Specifically, AIME and LiveBench-IF already usedeep, and we switch HotpotQA and PUPA fromfasttodeepat inference time.
Results:
Table R1. Four-family Qwen3-8B test evaluation with all specialists locked to deep (scores ; means from unrounded scores)
| Method / Profile Configuration | Specialist Profile | HotpotQA | AIME | PUPA | LB-IF | 4-Family Mean | vs. FPA-ALL |
|---|---|---|---|---|---|---|---|
| GEPA-FPA-ALL (paper Tables 2 & 8) | Global deep | 54.1 | 36.7 | 67.8 | 91.4 | 62.47 | — |
| Adaptive-GEPA (paper Tables 2 & 8) | deep: AIME, LB-IF; fast: HotpotQA, PUPA | 58.2 | 47.3 | 91.8 | 85.0 | 70.58 | +8.11 |
Adaptive-GEPA (All specialists deep) | All 4 active specialists deep | 58.3 | 49.3 | 89.8 | 85.6 | 70.76 | +8.29 |
Note: all four families were re-evaluated in this pass
- With all four active specialists running
deep,Adaptive-GEPAreaches 70.76 vs.GEPA-FPA-ALL's 62.47 (+8.29 points), closely matching its original 70.58 score (+8.11 points) and confirming thatAdaptive-GEPA's improvements come from our search structure and learned specialist library rather than execution-profile differences.
Thanks for your review (3/n)
Q4: Executed Adaptive-GEPA's Merge vs. Index-Based Merge on Identical Parent Pairs
"Appendix I.2 projects the aligned children's scores, whereas Table 12 includes a measured historical index child. Could you execute both aligned and index-based children on a few identical parent pairs from the main run, using the same scoring and capacity rules?"
In a two-stage controlled ablation on Qwen3-8B, Adaptive-GEPA's merge improves measured validation scores on identical parent pairs by +2.10 points across conflicting pairs (+1.20 overall) and raises the end-to-end test family mean from 65.10 (no merging) and 66.58 (index-based merging) to 70.58.
Experiment setup:
- Stage 1 (Same-parent merge execution): We executed both
Adaptive-GEPA's merge and index-based merge on 7 representative parent pairs from the main Qwen3-8B run—covering cross-branch slot collisions (P2, P4), inheriting a newly activated specialist slot (P3), avoiding description–program mismatch by keeping a slot's description and program paired (P7), and cases where both parents share the same slot indices without conflict (P1, P5, P6)—evaluating every child on the full validation set under the exact same scoring and capacity rules. - Stage 2 (End-to-end search): Across full Qwen3-8B optimization runs, we compare search without merging, with index-based merging, and with
Adaptive-GEPA's merging.
Results:
- In Stage 1 (same-parent merge execution),
Adaptive-GEPA's merge improves mean validation score from 64.60 to 65.80 (+1.20 points) across all 7 pairs, and from 64.78 to 66.88 (+2.10 points) across the 4 conflicting parent pairs (Table R2) while matching index-based merge when no slot conflict exists—reliably preventing specialist overwrites and mismatches when combining branches.
Table R2. Measured validation scores () of Adaptive-GEPA's merge vs. Index-based merge on identical parent pairs
| Parent Pair | Structural Difference Between Assembled Children | Adaptive-GEPA Merge | Index Merge | (Ours Index) |
|---|---|---|---|---|
| P1 | No slot conflict (identical child) | 64.35 | 64.35 | 0.00 |
| P2 | Slot collision: LB-IF in slot 4 (Parent A) vs. slot 1 (Parent B) | 68.42 | 63.54 | +4.88 |
| P3 | New specialist: Inheriting a newly activated LB-IF slot | 65.17 | 64.47 | +0.70 |
| P4 | Slot collision: LB-IF in slot 4 (Parent A) vs. slot 1 (Parent B) | 66.51 | 66.49 | +0.02 |
| P5 | No slot conflict (identical child) | 63.19 | 63.19 | 0.00 |
| P6 | No slot conflict (identical child) | 65.56 | 65.56 | 0.00 |
| P7 | Paired inheritance: Avoiding description–program mismatch in slot 5 | 67.42 | 64.63 | +2.79 |
| Mean (all 7 pairs) | 3 identical-child pairs + 4 conflicting parent pairs | 65.80 | 64.60 | +1.20 |
| Mean (4 conflicting parent pairs) | P2, P3, P4, P7 | 66.88 | 64.78 | +2.10 |
- In Stage 2 (end-to-end search), enabling index-based merging brings only a modest gain over no merging (65.10 66.58), whereas
Adaptive-GEPA's merging raises the test family mean to 70.58 (and from 58.18 59.27 63.50 excluding PUPA; Table R3), confirming thatAdaptive-GEPA's merging clearly outperforms both no merging and index-based merging.
Table R3. End-to-end Qwen3-8B comparison across merge configurations (scores )
| Merge Configuration | Validation Mean | Test Family Mean | Test Mean (excl. PUPA) |
|---|---|---|---|
| No merging | 65.73 | 65.10 | 58.18 |
| Index-based merging | 67.45 | 66.58 | 59.27 |
| Adaptive-GEPA's merging (Ours) | 74.03 | 70.58 | 63.50 |
Thanks for your review (1/n)
We sincerely thank you for your exceptionally thorough and thoughtful reading of both our main text and appendices, and we appreciate your recognition of our router–description–program representation, footprint-based branch alignment, and consistent gains over single-program search. Your detailed questions helped us clarify our experimental logs and strengthen our empirical analyses, and we address each question (Q1–Q5) and additional comment (A1–A3) point by point below.
Q1 & W1: GPT-4.1 Outage-Affected Entries (15–22) and Merge Count
Entries 15–22 were not repaired or re-evaluated, and the run accepted 20 merges compared with the intended cap of 15; importantly, (1) neither entries 15–22 nor the over-cap merges are in the ancestry of candidate 75, so the final GPT-4.1 program reported in our tables inherits no components from them, and (2) all merge evaluations were counted within the same fixed 18,000-call budget, adding no extra evaluation budget.
Q2 & W2: Test Monitoring Callback Curves
No components, prompts, hyperparameters, or protocol settings were tuned or changed based on inspecting the test callback curves. Within each run, all optimizer updates and candidate selection used Dfeedback and Dval exclusively (Appendix L); the callback logged scores every 5th accepted candidate for passive runtime monitoring, while hyperparameters were fixed as listed in Appendix B and the mechanism updates in Appendix I.1 were driven by validation-set and runtime crash diagnostics rather than test curves. Final evaluations are reported on the fixed held-out test splits of the four benchmarks without a separate post-development benchmark split; because no design or hyperparameter choices were made from the test callback curves, our final evaluation results were not affected by test-curve tuning.