- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Reflective Prompt Tuning (RPT), a diagnostic-driven framework for automated prompt optimization. RPT uses failure diagnosis on the full optimization set, failure clustering, and cross-iteration memory to drive prompt revision, and considers task score, calibration, and prompt length simultaneously when making the final selection. Experiments indicate that RPT offers advantages in both performance and prompt compactness across three tasks, with particularly notable results when using a GPT-5 optimizer. However, the current results do not allow for a fair disentanglement of RPT’s methodological contributions from the effects of higher computational budgets, different scaffolding, varying initial performance levels, or distinct prompt selection objectives.
Critical strengths
(1). The problem is practically significant; the issues of recurring failures and the utility of historical information in prompt optimization are indeed worthy of research. (2) . The methodological workflow of RPT is clearly articulated, and the interfaces between its modules are well-defined.
Critical weaknesses
(1). RPT’s diagnostic process involves extensive calls for target-model evaluation, critique, and clustering, whereas the baselines employ different optimization workflows and default budgets. (2). There are significant disparities in initial performance, making it impossible to attribute final performance gains solely to the optimization method. (3). The "w/o full-set" ablation removes both full-set coverage and clustering simultaneously, rendering the attribution of performance gains to individual components invalid. (4). RPT and GEPA-C utilize different protocols for calibration-based prompt selection. (5). RPT incurs high additional optimization costs; the actual token-based costs have not been sufficiently quantified.
Questions for authors
(1) Under a unified scaffold, an identical target-model call budget (or equivalent total budget), to what extent does RPT retain its advantage over ACE, GEPA, and MIPRO? (2) Is it possible to isolate and verify "full-set diagnosis" by varying only the failure sampling ratio, while keeping the clustering and reporting pipeline constant? (3) If GEPA-C were to employ the same calibration-aware selection objective as RPT, would RPT still maintain its advantage in terms of Brier score and ECE? (4) If the same historical prompts, metrics, and failure reports were incorporated into GEPA or other reflective baselines, would RPT still outperform them? (5) Are RPT’s results consistent across different target models, open-source models, and larger-scale tasks?
8 comments
Response to Reviewer 7dxo (2/4)
Weaknesses 1 and 5, Question 1 (continued)
Main findings
RPT makes more calls, but its calls are much cheaper. RPT makes 5.6–8.6 times as many calls as ACE, but each ACE request carries its growing playbook, whereas RPT sends only its compact prompt. Over the full runs, RPT sent 29.2 / 17.8 / 76.9 million input tokens versus ACE's 168.7 / 25.5 / 224.8 million (HotPotQA / LiveBench-Math / Formula). As a result, reaching the selected prompt costs RPT far less in target-model spend than ACE:
| GPT-4.1 target cost to reach selected prompt (USD, no caching) | RPT | ACE | GEPA | MIPRO |
|---|---|---|---|---|
| HotPotQA | 13.20 | 153.59 | 14.96 | ≥17.40 |
| LiveBench-Math | 20.96 | 39.78 | 2.83 | ≥14.79 |
| Formula | 42.26 | 83.73 | 4.46 | ≥11.03 |
RPT's cost is 0.09 / 0.53 / 0.50 times ACE's, and its development score at these checkpoints is within 2 points of ACE's on HotPotQA (+1.0) and Formula (−1.7). On HotPotQA, RPT is the cheapest of all four methods. GEPA has lower target cost on LiveBench-Math and Formula.
RPT finds its selected prompt early. RPT's selected prompts first appeared after only 34–38% of each run's total calls, so each run continued well past the point where it found the prompt it eventually selected. Currently, the optimizer LLM decides when to stop. An explicit stopping rule, such as stopping after a few iterations without improvement, could therefore reduce RPT's optimization cost substantially; testing such a rule remains future work.
Where RPT needs more budget. At ACE's final number of target evaluations, RPT matches ACE on HotPotQA (65.7% vs. 66.0% development score) but is still improving on Formula (73.3% vs. 88.3%) and LiveBench-Math (60.9% vs. 71.6%). On these tasks, RPT reaches its reported scores with more, but cheaper, target evaluations. This matches our new Appendix section "Performance After One Prompt Revision", where later iterations contribute most of the gain on these two tasks.
Held-out results on LiveBench-Math. In the new 732-query evaluation, the first checkpoints score alike (RPT 61.3% after 246 evaluations, ACE 62.2% after 260). RPT's iteration 6 reaches 72.8% after 1,476 evaluations, 4.4 points above ACE's selected prompt (68.4% after 512). RPT's own selected prompt scores 66.1%. All simultaneous 95% intervals for the RPT–ACE differences include zero, so neither method shows a reliable held-out advantage at these budgets.
RPT's prompts are much cheaper to serve. Across all 12 task–optimizer settings, ACE's selected prompt is at least 23 times larger than RPT's (in tokens). Values are USD per 1,000 responses, averaged over the queries after each method's first one; by then, the provider has cached the repeated prompt text and bills it at a lower rate:
| Serving cost | RPT | ACE | GEPA | MIPRO |
|---|---|---|---|---|
| HotPotQA | 5.27 | 75.58 | 5.53 | 9.72 |
| LiveBench-Math | 10.25 | 54.49 | 11.12 | 11.21 |
| Formula | 4.61 | 38.55 | 2.98 | 4.86 |
- HotPotQA: RPT has both the highest test score (68.4%, vs. 66.6% ACE, 64.4% GEPA, 66.8% MIPRO) and the lowest serving cost. ACE costs 14.3 times as much (saving $70.31 per 1,000 responses; paired 95% interval [$53.63, $94.73]).
- Formula: ACE scores 1.5 points higher (85.5% vs. 84.0%) but costs 8.4 times as much (saving $33.94; [$33.33, $34.63]). RPT scores 10.0 and 5.5 points above GEPA and MIPRO, at a serving cost between theirs.
- LiveBench-Math: The table excludes two questions on which RPT's response hit GPT-4.1's 32,768-token output limit (ACE's also did on one). On the rest, RPT saves $44.24 over ACE ([$29.62, $62.70]), and its differences from GEPA and MIPRO have intervals including zero. Including the two cut-off responses, the saving over ACE is $22.19, with an interval that includes zero.
- Without prompt caching, ACE costs 41, 20, and 8.7 times as much as RPT on HotPotQA, Formula, and LiveBench-Math.
The length penalty keeps prompts compact. With the penalty, RPT's selected Math and Formula prompts are 44% and 46% shorter than those selected without it. On HotPotQA, both rules select the same prompt.
Summary. RPT makes more calls than ACE, and on Formula and LiveBench-Math it needs more target evaluations to reach its reported scores. Because its prompts are compact, however, reaching its selected prompt costs less in target-model spend than ACE on all three tasks. With prompt caching, its serving cost is 8–14 times lower on HotPotQA and Formula, and 5.3 times lower on LiveBench-Math questions. Savings at serving time accumulate with every query. The saved records do not contain complete GPT-5 usage for every method, so we report target-model and serving costs rather than total bills and do not estimate a break-even point. We do not claim that RPT dominates at a matched call or dollar budget, and we revised Section 4 to state this explicitly.
Response to Reviewer 7dxo (3/4)
Weakness 2 (initial performance); Weakness 3 and Question 2 (full-set diagnosis)
Thank you for this suggestion. GEPA-C receives calibration diagnostics as auxiliary feedback but, unlike RPT, selects its final prompt by task score alone. To test whether RPT's calibration advantage comes from its final selection rule, we applied RPT's selection objective to GEPA-C's saved candidate prompts. We ran this test on LiveBench-Math, where GEPA-C's calibration differs most from RPT's (Figure 2), with the two strongest optimizers, GPT-5 and Gemini-3.1-Pro, and GPT-4.1 as the target model. Holding GEPA-C's generated candidates fixed, we selected among them with RPT's development-set selection objective (Eq. 2). Selection used only the development set; the test set was used only to evaluate the selected prompt. We compare the results with the RPT prompts reported in the paper for the same optimizers.
| Optimizer | GEPA-C candidate selected by RPT's objective | GEPA-C test Brier ↓ | GEPA-C test ECE ↓ | RPT test Brier ↓ | RPT test ECE ↓ |
|---|---|---|---|---|---|
| GPT-5 | Seed (candidate 0) | 0.543 | 0.618 | 0.182 | 0.207 |
| Gemini-3.1-Pro | Candidate 15 (unchanged) | 0.475 | 0.547 | 0.244 | 0.309 |
Under GPT-5, RPT's objective selects GEPA-C's seed rather than its originally selected candidate, and GEPA-C's test Brier rises from 0.519 to 0.543. Under Gemini-3.1-Pro, it selects the same candidate that GEPA-C originally selected. In both settings, GEPA-C's Brier and ECE remain 1.8–3.0 times RPT's. Thus, applying RPT's selection objective to GEPA-C's candidates does not close the calibration gap observed in Figure 2, so in these settings RPT's calibration advantage is not explained by its final selection rule.
Question 4, incorporating RPT's history into GEPA or other reflective baselines.
Thank you for this question. RPT's history is not a separate input that can be passed to another optimizer. Each report pairs a prompt with metrics on the full optimization set and with failure modes clustered from critiques of all its failures (Eq. 5). The memory then orders these reports along a single revision trajectory, and the optimizer uses it both to rewrite the prompt and to decide when to stop. The baselines make different design choices at each of these points (Appendix T in revised version):
- GEPA reflects on per-instance feedback from a minibatch and keeps a mutation only if its minibatch score improves. It also maintains a branching pool of candidates rather than one trajectory. Supplying RPT's reports would require evaluating every candidate on the full optimization set, replacing per-instance reflection with clustered diagnosis, and deciding which branch's history each mutation should see.
- ACE reflects on one sample at a time and merges new entries into a playbook that it deliberately does not rewrite. Using dataset-level reports would require reflecting over the full set and letting the curator rewrite or remove entries, which runs against its incremental design.
- MIPRO has no feedback stage that could receive the reports.
A baseline changed in all of these ways would no longer be GEPA or ACE. It would be RPT's diagnosis and memory inside a different search procedure, so RPT's contributions cannot be added to these methods without substantially changing them. We therefore tested whether these signals drive RPT's gains in two other ways. First, the ablations in Section 5.3 remove each signal from RPT in turn. Without memory (no earlier reports and no trajectory context ), task score falls on all three tasks, by 11 points on Formula, and the optimizer uses its full iteration budget on every task. Critiquing only sampled failures or only half of them lowers task score and worsens Brier score on every task. Second, we gave GEPA the parts of RPT that can be added without changing its search. GEPA-C receives calibration diagnostics as auxiliary feedback. RPT is still better calibrated on LiveBench-Math under every optimizer (Figure 2), and applying RPT's selection objective to GEPA-C's candidates does not close this gap (see our response to your previous question). Combining RPT's reports with GEPA's population search is an interesting direction for future work.
Response to Reviewer 7dxo (1/4)
Thank you for taking the time to review our paper and for the constructive comments. Your comments led to new experiments and analyses, summarized below. Several of your weaknesses correspond to your questions, so we answer them together.
Summary of changes
Weaknesses 1 and 5, Question 1: Optimization budget, cost, and a unified scaffold
Thank you for raising this. We agree that the methods were not budget-matched: each ran in its own native configuration, with its own optimization budget, so final scores alone do not establish an advantage at equal cost. To address this thoroughly, we revised our Cost Analysis appendix (new Appendix K) to examine cost from several angles. The call counts, budget comparisons, target-model cost estimates, and serving-cost measurements all use the GPT-5-optimizer runs of all four methods, with GPT-4.1 as the target model:
Main findings in the next response.