- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a two-stage prompt optimization process, where prompts obtained in the first phase (reasoning-guided search) are further optimized in a second phase using gradients (gradient-guided refinement).
Critical strengths
(1) Paper tackles the prompt optimization which is a timely problem.
Critical weaknesses
Major Weaknesses
(1) The paper’s novelty is incremental. The core idea is essentially to perform a second round of prompt optimization using gradients after a first round based on standard prompt optimizers.
(2) The experimental results are insufficient to support the authors’ thesis. The implicit assumption is that Phase 1 (reasoning-guided search) is insufficient for effective prompt optimization and therefore requires Phase 2 (gradient-guided refinement). However, for Phase 1, the authors evaluate only relatively outdated baselines (e.g., EvoPrompt). It is unclear whether the same conclusions hold for more recent prompt optimizers (e.g., [1-4]). With stronger optimizers, the second phase may become unnecessary.
(3) Even for these baselines, the experimental results are not very convincing. I averaged MATH-500 accuracy over 2 models and 5 Stage-1 settings provided by the authors (RPO-5, RPO-10, EvoPrompt-5, EvoPrompt-10, ETGPO-1). When a table entry was “-”, I used the corresponding Stage-1 result, since the authors explicitly state that “-” denotes no improvement on the development set and the Stage-1 prompt is retained. The results are:
| Refinement | Avg. accuracy |
|---|---|
| GradPO | 72.06 |
| GreaTer | 72.04 |
| GradPO-Prob | 71.99 |
| GreaTer-TG | 71.68 |
| LPO | 71.59 |
| No Refinement | 71.46 |
Therefore, GradPO performs comparably to GreaTer, while the authors explicitly report that GradPO requires 31,500 forward calls compared with only 7,500 for GreaTer. This leaves the reader unclear about the practical benefit of GradPO.
(4) There also appears to be a substantial risk of overfitting. The authors report that 25/29 cases passed development-set selection, but only 18/29 improved on the test set. Thus, 7 cases improved on the development set but not on the test set, suggesting possible development-set overfitting. This issue should be examined more carefully.
(5) The paper lacks important implementation details. For example, how do the authors compute gradient magnitudes? Do they use the L1 norm, L2 norm, max norm, or another measure?
Minor Weaknesses
(6) The paper completely omits reinforcement-learning-based approaches to prompt optimization, which are relevant to this work (e.g., [5-6]).
References
[1] Agrawal, Lakshya A., et al. "Gepa: Reflective prompt evolution can outperform reinforcement learning." ICLR'2026
[2] Agarwal, Eshaan, et al. "PromptWizard: Optimizing prompts via task-aware, feedback-driven self-evolution." ACL'2025
[3] Shi, Wenhang, et al. "No loss, no gain: Gated refinement and adaptive compression for prompt optimization." NeurIPS'2025
[4] Batorski, Paweł, and Paul Swoboda. "PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data." ACL'2026
[5] Deng, Mingkai, et al. "Rlprompt: Optimizing discrete text prompts with reinforcement learning." EMNLP'2022
[6] Kwon, Minchan, et al. "StablePrompt: Automatic prompt tuning using reinforcement learning for large language model." EMNLP'2024
Questions for authors
See weaknesses
5 comments
Part (2/2)
(our cont. response from last comment) Moreover, during rebuttal, we allowed sequential GreaTer 3–5 updates to match approximately the GradPO's candidate-scoring forward calls, while evaluating the same top five prompts on validation across FS-TACRED and MATH-500:
| Method | Validation improvements | Test improvements |
|---|---|---|
| GradPO | 18/20 | 11/20 |
| GreaTer | 9/20 | 5/20 |
Repeated GreaTer edits can also corrupt local text, producing “Okay for each following math problem'' and “Okayolve the following math problem'' from “Solve the following math problem.'' This shows that GradPO remains better than GreaTer under the same search budget, demonstrating the advantage of multi-span search over sequential token edits. We are also running FS-FewRel and will report it before the deadline.
-
There also appears to be a substantial risk of overfitting. The authors report that 25/29 cases passed development-set selection, but only 18/29 improved on the test set. Thus, 7 cases improved on the development set but not on the test set, suggesting possible development-set overfitting. This issue should be examined more carefully.
We respectfully disagree that the evidence shows “substantial” overfitting:
(a) Candidate prompts are selected using Equation (2), which penalizes variation across validation folds, directly reducing the risk of development-set overfitting (Lines 643–647).
(b) Second-stage hyperparameters are tuned only on RPO prompts at iterations 5 and 10 and transferred unchanged to EvoPrompt and ETGPO (Lines 307–309). Their successful transfer beyond the source-prompt family is inconsistent with substantial overfitting. Moreover, these are not random gains either: gradient selection also transfers more reliably than random selection: 18/29 versus 11/29 test improvements (Lines 414–419).
(c) Finally, test data are never used for prompt selection or hyperparameter tuning, and we report the complete development-to-test gap: 25/29 prompts pass development selection and 18/29 improve on test. Thus, 72% of selected refinements yield a positive test gain (Table 6), which does not indicate substantial overfitting.
-
The paper lacks important implementation details. For example, how do the authors compute gradient magnitudes? Do they use the L1 norm, L2 norm, max norm, or another measure?
We respectfully clarify that the paper already provides all necessary details: the algorithm and implementation procedure (Section 3.3, Algorithm 1; Lines 178–231), model and data settings, tuning protocol, and complete hyperparameter search spaces and selected values (Section 4 and Appendix B.1). The target model remains fixed, with no fine-tuning, and Appendix E provides the exact input, seed, and meta-prompts together with representative optimized prompts.
The only omitted detail raised by the reviewer is the norm used for token importance: we use the standard L2 norm for each token's gradient. We will make this explicit. We believe this is a minor notation omission rather than a missing method or experimental specification.
-
The paper completely omits reinforcement-learning-based approaches to prompt optimization, which are relevant to this work (e.g., [5-6]).
RL-based methods optimize prompts differently: they learn a policy from scalar rewards rather than reasoning over error feedback to make logical instruction-level revisions, as our first-stage methods do. Direct reward optimization can also produce less interpretable or gibberish-like prompts through reward hacking. Thus, they are closer to alternative standalone or first-stage optimizers than matched second-stage refiners.
Nevertheless, we are running StablePrompt (more recent than Rlprompt), both alone and followed by GradPO, and will report the results before the rebuttal deadline. Our current evaluation already covers three diverse first-stage optimizers and five second-stage refiners across two models and three datasets. We will also cite and discuss RLPrompt and StablePrompt in the revised paper.
Overall, we hope our clarification of the contributions, matched-budget experiment, cluster-aware analysis, and implementation details address the reviewer’s concerns; GEPA and StablePrompt evaluations are also in progress. The resulting evidence and revisions can be readily incorporated into the paper.
Below, we report the additional GEPA results discussed above.
We additionally evaluated GEPA as a recent reflection-based first-stage optimizer at 5, 10, and 20 mutation iterations, followed by GradPO, LPO, or GreaTer with a search budget similar to GradPO’s. GEPA reports 1,839 rollouts per mathematical benchmark, corresponding to approximately 7–10 mutation iterations under our setup. Thus, our 20-iteration run already exceeds this budget, while FS-TACRED requires substantially more executions per validation pass. A dash indicates that the refinement did not improve validation performance and was not evaluated on the test set. Bold marks a higher mean test score than the corresponding GEPA prompt. FS-TACRED reports micro-F1, while MATH-500 reports accuracy.
| Dataset | Model | GEPA source | GEPA | +GradPO | +LPO | +GreaTer (similar budget) |
|---|---|---|---|---|---|---|
| FS-TACRED | Qwen3-4B | GEPA-5 | 21.4 ± 1.23 | 20.9 ± 1.50 | 18.0 ± 1.18 | 20.1 ± 1.43 |
| FS-TACRED | Qwen3-4B | GEPA-10 | 29.0 ± 2.02 | 29.8 ± 1.91 | - | 29.9 ± 2.60 |
| FS-TACRED | Qwen3-4B | GEPA-20 | 25.2 ± 0.88 | - | - | 24.0 ± 1.27 |
| FS-TACRED | Gemma3-4B | GEPA-5 | 10.2 ± 0.66 | 11.7 ± 0.32 | 9.7 ± 0.48 | 9.9 ± 0.59 |
| FS-TACRED | Gemma3-4B | GEPA-10 | 10.4 ± 0.61 | - | - | 10.4 ± 0.59 |
| FS-TACRED | Gemma3-4B | GEPA-20 | 10.5 ± 0.41 | 11.2 ± 0.41 | - | 9.9 ± 0.24 |
| MATH-500 | Qwen3-4B | GEPA-5 | 72.8 ± 1.29 | 73.2 ± 0.84 | - | 72.6 ± 0.49 |
| MATH-500 | Qwen3-4B | GEPA-10 | 71.6 ± 0.48 | - | - | 73.3 ± 1.29 |
| MATH-500 | Qwen3-4B | GEPA-20 | 76.2 ± 0.80 | 76.0 ± 0.26 | - | 76.5 ± 0.81 |
| MATH-500 | Gemma3-4B | GEPA-5 | 63.0 ± 1.16 | 64.5 ± 1.32 | - | - |
| MATH-500 | Gemma3-4B | GEPA-10 | 62.1 ± 1.80 | - | - | - |
| MATH-500 | Gemma3-4B | GEPA-20 | 67.0 ± 1.51 | 66.6 ± 0.82 | - | 65.4 ± 0.95 |
Improvement frequency
| Second-stage method | Dev improvements | Test improvements |
|---|---|---|
| GradPO | 8/12 | 5/12 |
| LPO | 2/12 | 0/12 |
| Exact-budget GreaTer | 10/12 | 3/12 |
GradPO achieves the highest test-improvement frequency, improving 5/12 GEPA prompts, compared with 3/12 for GreaTer using a similar search budget and 0/12 for LPO. Although GreaTer passes development selection more often, many of its refinements do not transfer to the test set. These results show that GradPO effectively refines prompts produced by the recent GEPA optimizer and outperforms the other refiners.
We appreciate Reviewer CGUY’s constructive comments and respond to the comments below.
Response to weaknesses
We respectfully disagree that the contribution is merely a second round of optimization. The paper makes two distinct contributions:
(1) a two-stage framework that combines broad reasoning-based search with targeted gradient-based refinement, motivated by LLM sensitivity to small wording choices left unresolved by broad semantic edits (Lines 40–53 and 94–96); and
(2) GradPO, a new local optimizer that couples gradient-guided multi-span selection with bidirectional context-aware replacement using only the target model (Lines 98–101 and 178–211). Table 1 summarizes GradPO's advantages over GreaTer and LPO. Although simple to apply, one GradPO step consistently improves prompts already optimized by strong first-stage methods, and this effectiveness is a major practical contribution. Empirically, GradPO improves 18/29 such prompts and is best in 10 blocks, six more improvements and four more wins than the next-best alternative (Lines 398–410).
We respectfully clarify that our first-stage study is not limited to EvoPrompt (ICLR 2024). We also use feedback-driven RPO and taxonomy-guided ETGPO (arXiv 2026), together covering diverse strategies: iterative reflection and revision, evolutionary search, and structured error-based revision (Lines 283–290). Newer methods cited by the reviewer, including GEPA, PromptWizard, and GRACE, similarly use task feedback or reflection to make broad instruction-level revisions; RPO represents this general search family (Lines 283–285). PIAST instead primarily optimizes in-context examples and is therefore less directly aligned with our instruction-optimization setting.
More importantly, the second stage is needed because reasoning-based optimizers make broad semantic and logical changes, while target LLMs remain sensitive to small wording choices that these searches can leave unresolved (Lines 40–49). Table 7 directly tests longer first-stage search: one GradPO step after iteration 10 still improves 7/8 settings and outperforms the best prompt found over 20 first-stage iterations in 5/8 settings (Lines 421–425). These results empirically show that local refinement remains useful even after longer first-stage search.
To directly test the reviewer's concern with a newer first-stage optimizer, we are running GEPA alone and GEPA followed by GradPO under the same setting, and we will share the results before the rebuttal deadline.
We respectfully disagree with this interpretation: this presentation considers only MATH-500 and also replaces missing second-stage results with unchanged first-stage scores, so it does not measure the actual gains of individual 2nd-stage refiners. Across all three datasets, averaging only valid 2nd-stage outputs, we get the following results:
GradPO has the largest mean gain over its corresponding first-stage prompts. GradPO also achieves 11 statistically significant gains, compared with 5 for GreaTer (Tables 2–3; Lines 398–410).
... (cont. in next comment)
Part(1/2)