acceptodds
Under review as a conference paper at ICLR 2027

Two-Stage Prompt Optimization: From Reasoning-Guided Search to Gradient-Guided Refinement

Abstract

Existing automatic prompt optimization methods often focus on either broad semantic search or fine-grained local refinement, limiting their ability to benefit from both. We propose a two-stage framework that combines complementary reasoning-based and gradient-based prompt optimization, and introduce GradPO as its second-stage refiner. The first stage can use any general reasoning-based optimizer to make broad prompt improvements based on feedback. In the second stage, GradPO uses target-model gradients to select multiple high-impact spans and combines short, context-aware replacements through beam search, without requiring a separate optimizer model. Experiments on FS-TACRED, FS-FewRel, and MATH-500, spanning relation extraction and mathematical reasoning, show that these two forms of optimization complement each other: local refinement usually further improves prompts found by the first stage. Across the relation extraction tasks, GradPO provides the most consistent improvements, achieving state-of-the-art performance on FS-TACRED with Qwen3-4B while remaining competitive on FS-FewRel. On MATH-500, our GradPO variants improve first-stage prompts more frequently than the alternative refiners, further showing that our approach is effective across both reasoning and non-reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automatic prompt optimization, gradient-guided prompt optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CGUYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage prompt optimization process, where prompts obtained in the first phase (reasoning-guided search) are further optimized in a second phase using gradients (gradient-guided refinement).

Critical strengths

(1) Paper tackles the prompt optimization which is a timely problem.

Critical weaknesses

Major Weaknesses

(1) The paper’s novelty is incremental. The core idea is essentially to perform a second round of prompt optimization using gradients after a first round based on standard prompt optimizers.

(2) The experimental results are insufficient to support the authors’ thesis. The implicit assumption is that Phase 1 (reasoning-guided search) is insufficient for effective prompt optimization and therefore requires Phase 2 (gradient-guided refinement). However, for Phase 1, the authors evaluate only relatively outdated baselines (e.g., EvoPrompt). It is unclear whether the same conclusions hold for more recent prompt optimizers (e.g., [1-4]). With stronger optimizers, the second phase may become unnecessary.

(3) Even for these baselines, the experimental results are not very convincing. I averaged MATH-500 accuracy over 2 models and 5 Stage-1 settings provided by the authors (RPO-5, RPO-10, EvoPrompt-5, EvoPrompt-10, ETGPO-1). When a table entry was “-”, I used the corresponding Stage-1 result, since the authors explicitly state that “-” denotes no improvement on the development set and the Stage-1 prompt is retained. The results are:

RefinementAvg. accuracy
GradPO72.06
GreaTer72.04
GradPO-Prob71.99
GreaTer-TG71.68
LPO71.59
No Refinement71.46

Therefore, GradPO performs comparably to GreaTer, while the authors explicitly report that GradPO requires 31,500 forward calls compared with only 7,500 for GreaTer. This leaves the reader unclear about the practical benefit of GradPO.

(4) There also appears to be a substantial risk of overfitting. The authors report that 25/29 cases passed development-set selection, but only 18/29 improved on the test set. Thus, 7 cases improved on the development set but not on the test set, suggesting possible development-set overfitting. This issue should be examined more carefully.

(5) The paper lacks important implementation details. For example, how do the authors compute gradient magnitudes? Do they use the L1 norm, L2 norm, max norm, or another measure?

Minor Weaknesses

(6) The paper completely omits reinforcement-learning-based approaches to prompt optimization, which are relevant to this work (e.g., [5-6]).

References

[1] Agrawal, Lakshya A., et al. "Gepa: Reflective prompt evolution can outperform reinforcement learning." ICLR'2026

[2] Agarwal, Eshaan, et al. "PromptWizard: Optimizing prompts via task-aware, feedback-driven self-evolution." ACL'2025

[3] Shi, Wenhang, et al. "No loss, no gain: Gated refinement and adaptive compression for prompt optimization." NeurIPS'2025

[4] Batorski, Paweł, and Paul Swoboda. "PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data." ACL'2026

[5] Deng, Mingkai, et al. "Rlprompt: Optimizing discrete text prompts with reinforcement learning." EMNLP'2022

[6] Kwon, Minchan, et al. "StablePrompt: Automatic prompt tuning using reinforcement learning for large language model." EMNLP'2024

Questions for authors

See weaknesses

5 comments

We appreciate Reviewer CGUY’s constructive comments and respond to the comments below.

Response to weaknesses

  1. The paper’s novelty is incremental. The core idea is essentially to perform a second round of prompt optimization using gradients after a first round based on standard prompt optimizers.

We respectfully disagree that the contribution is merely a second round of optimization. The paper makes two distinct contributions:

(1) a two-stage framework that combines broad reasoning-based search with targeted gradient-based refinement, motivated by LLM sensitivity to small wording choices left unresolved by broad semantic edits (Lines 40–53 and 94–96); and

(2) GradPO, a new local optimizer that couples gradient-guided multi-span selection with bidirectional context-aware replacement using only the target model (Lines 98–101 and 178–211). Table 1 summarizes GradPO's advantages over GreaTer and LPO. Although simple to apply, one GradPO step consistently improves prompts already optimized by strong first-stage methods, and this effectiveness is a major practical contribution. Empirically, GradPO improves 18/29 such prompts and is best in 10 blocks, six more improvements and four more wins than the next-best alternative (Lines 398–410).

  1. The experimental results are insufficient to support the authors’ thesis. The implicit assumption is that Phase 1 (reasoning-guided search) is insufficient for effective prompt optimization and therefore requires Phase 2 (gradient-guided refinement). However, for Phase 1, the authors evaluate only relatively outdated baselines (e.g., EvoPrompt). It is unclear whether the same conclusions hold for more recent prompt optimizers (e.g., [1-4]). With stronger optimizers, the second phase may become unnecessary.

We respectfully clarify that our first-stage study is not limited to EvoPrompt (ICLR 2024). We also use feedback-driven RPO and taxonomy-guided ETGPO (arXiv 2026), together covering diverse strategies: iterative reflection and revision, evolutionary search, and structured error-based revision (Lines 283–290). Newer methods cited by the reviewer, including GEPA, PromptWizard, and GRACE, similarly use task feedback or reflection to make broad instruction-level revisions; RPO represents this general search family (Lines 283–285). PIAST instead primarily optimizes in-context examples and is therefore less directly aligned with our instruction-optimization setting.

More importantly, the second stage is needed because reasoning-based optimizers make broad semantic and logical changes, while target LLMs remain sensitive to small wording choices that these searches can leave unresolved (Lines 40–49). Table 7 directly tests longer first-stage search: one GradPO step after iteration 10 still improves 7/8 settings and outperforms the best prompt found over 20 first-stage iterations in 5/8 settings (Lines 421–425). These results empirically show that local refinement remains useful even after longer first-stage search.

To directly test the reviewer's concern with a newer first-stage optimizer, we are running GEPA alone and GEPA followed by GradPO under the same setting, and we will share the results before the rebuttal deadline.

  1. Even for these baselines, the experimental results are not very convincing. I averaged MATH-500 accuracy over 2 models and 5 Stage-1 settings provided by the authors (RPO-5, RPO-10, EvoPrompt-5, EvoPrompt-10, ETGPO-1). When a table entry was “-”, I used the corresponding Stage-1 result, since the authors explicitly state that “-” denotes no improvement on the development set and the Stage-1 prompt is retained. The results are [Table from the comment]

Therefore, GradPO performs comparably to GreaTer, while the authors explicitly report that GradPO requires 31,500 forward calls compared with only 7,500 for GreaTer. This leaves the reader unclear about the practical benefit of GradPO.

We respectfully disagree with this interpretation: this presentation considers only MATH-500 and also replaces missing second-stage results with unchanged first-stage scores, so it does not measure the actual gains of individual 2nd-stage refiners. Across all three datasets, averaging only valid 2nd-stage outputs, we get the following results:

MethodMean gain over Stage 1
GradPO+0.89
GreaTer+0.62
LPO+0.56
GradPO-Prob+0.47
GreaTer-TG+0.08

GradPO has the largest mean gain over its corresponding first-stage prompts. GradPO also achieves 11 statistically significant gains, compared with 5 for GreaTer (Tables 2–3; Lines 398–410).

... (cont. in next comment)

Part(1/2)

Part (2/2)

(our cont. response from last comment) Moreover, during rebuttal, we allowed sequential GreaTer 3–5 updates to match approximately the GradPO's candidate-scoring forward calls, while evaluating the same top five prompts on validation across FS-TACRED and MATH-500:

MethodValidation improvementsTest improvements
GradPO18/2011/20
GreaTer9/205/20

Repeated GreaTer edits can also corrupt local text, producing “Okay for each following math problem'' and “Okayolve the following math problem'' from “Solve the following math problem.'' This shows that GradPO remains better than GreaTer under the same search budget, demonstrating the advantage of multi-span search over sequential token edits. We are also running FS-FewRel and will report it before the deadline.

  1. There also appears to be a substantial risk of overfitting. The authors report that 25/29 cases passed development-set selection, but only 18/29 improved on the test set. Thus, 7 cases improved on the development set but not on the test set, suggesting possible development-set overfitting. This issue should be examined more carefully.

We respectfully disagree that the evidence shows “substantial” overfitting:

(a) Candidate prompts are selected using Equation (2), which penalizes variation across validation folds, directly reducing the risk of development-set overfitting (Lines 643–647).

(b) Second-stage hyperparameters are tuned only on RPO prompts at iterations 5 and 10 and transferred unchanged to EvoPrompt and ETGPO (Lines 307–309). Their successful transfer beyond the source-prompt family is inconsistent with substantial overfitting. Moreover, these are not random gains either: gradient selection also transfers more reliably than random selection: 18/29 versus 11/29 test improvements (Lines 414–419).

(c) Finally, test data are never used for prompt selection or hyperparameter tuning, and we report the complete development-to-test gap: 25/29 prompts pass development selection and 18/29 improve on test. Thus, 72% of selected refinements yield a positive test gain (Table 6), which does not indicate substantial overfitting.

  1. The paper lacks important implementation details. For example, how do the authors compute gradient magnitudes? Do they use the L1 norm, L2 norm, max norm, or another measure?

We respectfully clarify that the paper already provides all necessary details: the algorithm and implementation procedure (Section 3.3, Algorithm 1; Lines 178–231), model and data settings, tuning protocol, and complete hyperparameter search spaces and selected values (Section 4 and Appendix B.1). The target model remains fixed, with no fine-tuning, and Appendix E provides the exact input, seed, and meta-prompts together with representative optimized prompts.

The only omitted detail raised by the reviewer is the norm used for token importance: we use the standard L2 norm for each token's gradient. We will make this explicit. We believe this is a minor notation omission rather than a missing method or experimental specification.

  1. The paper completely omits reinforcement-learning-based approaches to prompt optimization, which are relevant to this work (e.g., [5-6]).

RL-based methods optimize prompts differently: they learn a policy from scalar rewards rather than reasoning over error feedback to make logical instruction-level revisions, as our first-stage methods do. Direct reward optimization can also produce less interpretable or gibberish-like prompts through reward hacking. Thus, they are closer to alternative standalone or first-stage optimizers than matched second-stage refiners.

Nevertheless, we are running StablePrompt (more recent than Rlprompt), both alone and followed by GradPO, and will report the results before the rebuttal deadline. Our current evaluation already covers three diverse first-stage optimizers and five second-stage refiners across two models and three datasets. We will also cite and discuss RLPrompt and StablePrompt in the revised paper.

Overall, we hope our clarification of the contributions, matched-budget experiment, cluster-aware analysis, and implementation details address the reviewer’s concerns; GEPA and StablePrompt evaluations are also in progress. The resulting evidence and revisions can be readily incorporated into the paper.

Below, we report the additional GEPA results discussed above.

We additionally evaluated GEPA as a recent reflection-based first-stage optimizer at 5, 10, and 20 mutation iterations, followed by GradPO, LPO, or GreaTer with a search budget similar to GradPO’s. GEPA reports 1,839 rollouts per mathematical benchmark, corresponding to approximately 7–10 mutation iterations under our setup. Thus, our 20-iteration run already exceeds this budget, while FS-TACRED requires substantially more executions per validation pass. A dash indicates that the refinement did not improve validation performance and was not evaluated on the test set. Bold marks a higher mean test score than the corresponding GEPA prompt. FS-TACRED reports micro-F1, while MATH-500 reports accuracy.

DatasetModelGEPA sourceGEPA+GradPO+LPO+GreaTer (similar budget)
FS-TACREDQwen3-4BGEPA-521.4 ± 1.2320.9 ± 1.5018.0 ± 1.1820.1 ± 1.43
FS-TACREDQwen3-4BGEPA-1029.0 ± 2.0229.8 ± 1.91-29.9 ± 2.60
FS-TACREDQwen3-4BGEPA-2025.2 ± 0.88--24.0 ± 1.27
FS-TACREDGemma3-4BGEPA-510.2 ± 0.6611.7 ± 0.329.7 ± 0.489.9 ± 0.59
FS-TACREDGemma3-4BGEPA-1010.4 ± 0.61--10.4 ± 0.59
FS-TACREDGemma3-4BGEPA-2010.5 ± 0.4111.2 ± 0.41-9.9 ± 0.24
MATH-500Qwen3-4BGEPA-572.8 ± 1.2973.2 ± 0.84-72.6 ± 0.49
MATH-500Qwen3-4BGEPA-1071.6 ± 0.48--73.3 ± 1.29
MATH-500Qwen3-4BGEPA-2076.2 ± 0.8076.0 ± 0.26-76.5 ± 0.81
MATH-500Gemma3-4BGEPA-563.0 ± 1.1664.5 ± 1.32--
MATH-500Gemma3-4BGEPA-1062.1 ± 1.80---
MATH-500Gemma3-4BGEPA-2067.0 ± 1.5166.6 ± 0.82-65.4 ± 0.95

Improvement frequency

Second-stage methodDev improvementsTest improvements
GradPO8/125/12
LPO2/120/12
Exact-budget GreaTer10/123/12

GradPO achieves the highest test-improvement frequency, improving 5/12 GEPA prompts, compared with 3/12 for GreaTer using a similar search budget and 0/12 for LPO. Although GreaTer passes development selection more often, many of its refinements do not transfer to the test set. These results show that GradPO effectively refines prompts produced by the recent GEPA optimizer and outperforms the other refiners.

Reviewer_NpAUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a two-stage framework for natural-language prompt optimization. A reasoning-based optimizer first produces broad revisions to an initial prompt, after which the proposed GradPO performs gradient-guided local refinement. GradPO uses target-model gradients to identify high-impact spans, generates context-aware replacements with the target model itself, and combines these replacements through beam search using a task-loss and fluency objective. The paper also investigates a probability-based replacement variant, GradPO-Prob. The second stage requires gradient access to the target model but does not update its parameters or require a separate optimizer model.

Experiments cover two 4B target models, three first-stage optimizers, and three datasets spanning episodic few-shot relation extraction and mathematical reasoning. The authors report that GradPO improves 18 of 29 valid first-stage prompt configurations, with 11 statistically significant improvements under their testing procedure. The central contribution is the proposed complementarity between broad reasoning-based search and targeted numerical-gradient refinement.

Critical strengths

Strength1. A practically relevant refinement setting with a concrete algorithmic design.

The paper addresses a useful question: whether an already optimized natural-language prompt can benefit from a different form of optimization rather than simply extending the original search. GradPO provides a concrete implementation of this idea through gradient-selected spans, context-aware replacement generation, and compositional beam search. Its use of the target model alone in the second stage makes it a potentially useful addition to existing pipelines when gradient access is available.

Strength2. The experiments contain informative controls and meaningful positive results.

All second-stage methods start from the same first-stage prompt within each configuration. The paper also includes a random-span control, an analysis of longer first-stage search, and experiments that apply gradient-based optimization directly to the initial prompt. These controls provide useful evidence beyond isolated best-case examples. Some improvements are meaningful: for example, GradPO raises Qwen3-4B from 26.5 to 29.0 F1 on FS-TACRED after RPO-5, and from 79.4% to 81.2% accuracy on MATH-500 after RPO-10. The remaining issue is whether these benefits persist under stronger experimental controls, rather than whether the paper demonstrates any benefit at all.

Critical weaknesses

Weakness1. The incremental benefit over closely related optimizers is not sufficiently isolated, and the competitive positioning needs to be more precise.

Several important ingredients have direct precedents. Local Prompt Optimization (LPO) [1] already identifies restricted editing regions and integrates localized revisions into general prompt-optimization procedures. GReaTer [2] already uses numerical gradients and target-model-generated token candidates for discrete prompt optimization without requiring a separate large optimizer. These methods do not subsume GradPO: in particular, LPO does not use numerical gradients, and GReaTer does not perform the same multi-span compositional search. Nevertheless, the contribution should be established specifically for the combination of gradient-based span selection, context-aware replacements, and beam search, rather than for local refinement or target-model self-optimization in general.

The current comparison does not cleanly establish that contribution. Both GReaTer variants are restricted to one single-token refinement step, whereas one GradPO iteration explores combinations of several span edits. Appendix B.6 reports 31,500 candidate-scoring forward calls for the largest relation-extraction GradPO configuration, compared with 7,500 for GReaTer. This is a 4.2-fold difference in candidate-scoring calls, not in total runtime; I acknowledge that both methods subsequently incur the same large validation cost. However, equal iteration counts do not constitute equal local-search budgets. Since the original GReaTer is an iterative coordinate-update procedure [2], a stronger control would allow multiple GReaTer updates under a matched overall refinement budget, including a matched validation budget. Performance–computation comparisons are particularly important for search-based methods [4]. The longer-run experiments in Tables 7 and 10 are useful, but they do not provide this comparison from the same first-stage prompt.

The paper’s own results also require a more qualified account of its advantage over LPO [1]. In Table 2, after the same RPO-10 prompt, LPO obtains 31.9 F1 on Qwen3-4B/FS-TACRED, compared with 28.9 for GradPO. On Gemma3-4B/FS-FewRel after RPO-5, LPO obtains 37.5 F1, compared with 32.7 for GradPO. LPO uses a larger optimizer in these experiments, so these results should be interpreted as a quality–resource trade-off rather than unconditional dominance. Nevertheless, the SOTA discussion should acknowledge the stronger internal LPO result, and the frequency of positive improvements should not substitute for reporting effect sizes and the magnitude of regressions.

Finally, GEPA [3] is a relevant omitted contemporary optimizer: it combines natural-language reflection over execution traces with Pareto-based prompt evolution and reports strong results against MIPROv2 in its own evaluation. A budget-controlled comparison of GEPA versus GEPA followed by GradPO would help test whether the proposed complementarity extends to another competitive reflection-based search procedure. This omission is not evidence that GEPA would outperform GradPO on the present benchmarks, nor is it alone grounds for rejection; it limits the breadth of the current claim.

Weakness2. The evidence for optimizer-level reliability does not adequately separate optimization randomness from evaluation variability.

The five reported runs do not clearly correspond to independent repetitions of the optimization procedure. For relation extraction, Tables 4 and 5 describe variation over five test splits. For MATH-500, Appendix B.5 describes five decoding seeds applied to the same 500 problems for each selected prompt. These measurements are useful for evaluating a particular optimized prompt, but they do not establish how reliably the optimizer produces an improvement when training-example sampling, replacement generation, and prompt selection are repeated. The distinction between evaluating a particular artifact and evaluating the stochastic procedure that generates it is directly relevant here [5]. The manuscript should explicitly state the number of independent optimization runs and report the resulting distribution of gains.

The significance analysis also needs to respect the dependency structure of the evaluation. Appendix B.5 resamples 2,500 individual problem–seed outputs for MATH-500 and 150,000 individual query predictions for relation extraction. Outputs for the same mathematical problem share problem-specific difficulty, while queries within an episode share its support context and candidate relations. Resampling these observations as flat collections does not preserve those groupings and may understate uncertainty when generalizing to new problems or episodes. A paired cluster-aware analysis, together with uncertainty across independent optimization runs and an explicit treatment of multiple comparisons, would provide stronger support [5,6].

Consequently, the reported counts of 18 improvements and 11 significant gains are useful descriptive evidence, but they do not yet establish that GradPO is a reliably better optimization procedure. This matters because consistency, rather than uniform superiority in individual settings, is the paper’s principal empirical justification.

References

[1] Jain, Y., and Chowdhary, V. (2025). Local Prompt Optimization. NAACL 2025, pp. 75–81. https://aclanthology.org/2025.naacl-short.7/

[2] Das, S. S. S., et al. (2025). GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers. ICLR 2025. https://arxiv.org/abs/2412.09722

[3] Agrawal, L. A., et al. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026. https://arxiv.org/abs/2507.19457

[4] Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N. A. (2019). Show Your Work: Improved Reporting of Experimental Results. EMNLP-IJCNLP 2019. https://aclanthology.org/D19-1224/

[5] Sellam, T., et al. (2022). The MultiBERTs: BERT Reproductions for Robustness Analysis. ICLR 2022. https://openreview.net/forum?id=K0E_F0gFDgA

[6] Dror, R., Baumer, G., Shlomov, S., and Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. ACL 2018, pp. 1383–1392. https://aclanthology.org/P18-1128/

Questions for authors

Question1. Additional evidence needed for the main comparison. Starting from the same first-stage prompt, does GradPO retain its advantage when GReaTer/GReaTer-TG are allowed multiple updates under a matched overall refinement budget and the same validation budget? Please also apply the same protected-term constraints across methods and report actual runtime or another comparable computation measure, rather than iteration counts alone.

Question2. Clarification and statistical reanalysis.
How many independent prompt-optimization runs were performed for each configuration, as distinct from test splits and decoding seeds? Do the main conclusions, including the counts of significant improvements, remain unchanged under a paired analysis that preserves problem-level or episode-level groupings and accounts for multiple comparisons?

Question3. Implementation clarification. Algorithm 1 selects the refined prompt by minimizing validation loss, whereas Appendix B.1 states that retained candidates are selected using Equation (2), which penalizes variability in the task metric. Which rule produced the reported results? Are protected terms and edit boundaries enforced programmatically or only through prompting? This is relevant because Appendix B.1 protects “yes” for GradPO, while Appendix E.6.1 shows a GReaTer-TG candidate changing the answer label from “yes” to “your.”

6 comments

We sincerely thank Reviewer NpAU for their valuable feedback. We address their feedback in detail below:

Response to weaknesses

Weakness1. The incremental benefit over closely related optimizers is not sufficiently isolated, and the competitive positioning needs to be more precise.

Several important ingredients have direct precedents. Local Prompt Optimization (LPO) [1] already identifies restricted editing regions and integrates localized revisions into general prompt-optimization procedures. GReaTer [2] already uses numerical gradients and target-model-generated token candidates for discrete prompt optimization without requiring a separate large optimizer. These methods do not subsume GradPO: in particular, LPO does not use numerical gradients, and GReaTer does not perform the same multi-span compositional search.

GreaTer and LPO are logical local-refinement baselines precisely because they use different approaches: GreaTer provides gradient-guided single-token editing, while LPO uses non-gradient, LLM-guided span editing (Lines 242–259). GradPO instead performs gradient-guided multi-span editing (Lines 178–211). We adapt both baselines as second-stage refiners starting from the same first-stage prompts for fair comparison (Lines 238–240 and 367–370). We will clarify this more in the paper.

(cont.) Nevertheless, the contribution should be established specifically for the combination of gradient-based span selection, context-aware replacements, and beam search, rather than for local refinement or target-model self-optimization in general.

Respectfully, our submission presents two distinct contributions. First, the two-stage framework introduces local refinement after reasoning-based global search (Lines 50–53 and 94–96). This stage is needed because reasoning-based optimizers make broad semantic changes, while LLMs remain sensitive to small wording choices that gradient-based local refinement can target (Lines 40–49). Second, GradPO is an independent method contribution that couples target-gradient adaptive-span selection, context-aware replacement, and multi-span beam composition (Lines 98–101 and 178–211). These should therefore be considered separately: one is the two-stage framework, and the other is the complete GradPO procedure.

The current comparison does not cleanly establish that contribution. Both GReaTer variants are restricted to one single-token refinement step, whereas one GradPO iteration explores combinations of several span edits. Appendix B.6 reports 31,500 candidate-scoring forward calls for the largest relation-extraction GradPO configuration, compared with 7,500 for GReaTer. This is a 4.2-fold difference in candidate-scoring calls, not in total runtime; I acknowledge that both methods subsequently incur the same large validation cost. However, equal iteration counts do not constitute equal local-search budgets. Since the original GReaTer is an iterative coordinate-update procedure [2], a stronger control would allow multiple GReaTer updates under a matched overall refinement budget, including a matched validation budget. Performance–computation comparisons are particularly important for search-based methods [4]. The longer-run experiments in Tables 7 and 10 are useful, but they do not provide this comparison from the same first-stage prompt.

Thank you for this comment. During rebuttal, we ran experiments that allow GreaTer 3–5 sequential updates to match GradPO's candidate-scoring forward calls, then evaluated the top five prompts using the same validation budget across FS-TACRED and MATH-500:

MethodValidation improvementsTest improvements
GradPO18/2011/20
GreaTer9/205/20

Repeated GreaTer edits can also corrupt local text, producing “Okay for each following math problem'' and “Okayolve the following math problem'' from “Solve the following math problem.'' This shows that GradPO’s multi-span search outperforms GreaTer’s sequential token editing under the same budget. We are also running FS-FewRel and will report it before the deadline.

The paper’s own results also require a more qualified account of its advantage over LPO [1]. In Table 2, after the same RPO-10 prompt, LPO obtains 31.9 F1 on Qwen3-4B/FS-TACRED, compared with 28.9 for GradPO. On Gemma3-4B/FS-FewRel after RPO-5, LPO obtains 37.5 F1, compared with 32.7 for GradPO. LPO uses a larger optimizer in these experiments, so these results should be interpreted as a quality–resource trade-off rather than unconditional dominance.

Although LPO is higher in these two blocks, across configurations, performance should be judged by consistency rather than a few occasional high scores: GradPO improves 18/29 prompts and is best in 10 blocks, whereas LPO improves 9/29 and is best in 5 (Lines 399–410). Table 1 also summarizes GradPO's design and resource advantages over LPO. We will clarify this distinction further.

Part(1/4)

Part (2/4)

(cont.) Nevertheless, the SOTA discussion should acknowledge the stronger internal LPO result, and the frequency of positive improvements should not substitute for reporting effect sizes and the magnitude of regressions.

Our SOTA claim is specific to GradPO on FS-TACRED with Qwen3-4B, where it reaches 29.0 F1 and exceeds the prior published best of 27.6 (Lines 427–435). We do not claim that GradPO is highest in every individual block; we claim that it is the most consistent refiner across configurations (Lines 405–410).

Tables 2–3 already report the mean and standard deviation for every evaluated result. We are not sure what the reviewer meant by “magnitude of regressions.”

Finally, GEPA [3] is a relevant omitted contemporary optimizer: it combines natural-language reflection over execution traces with Pareto-based prompt evolution and reports strong results against MIPROv2 in its own evaluation. A budget-controlled comparison of GEPA versus GEPA followed by GradPO would help test whether the proposed complementarity extends to another competitive reflection-based search procedure. This omission is not evidence that GEPA would outperform GradPO on the present benchmarks, nor is it alone grounds for rejection; it limits the breadth of the current claim.

We would like to clarify that GEPA (and MIPROv2) is primarily motivated by compound LM systems that contain multiple prompts to optimize. When applied to our single-prompt setting, GEPA's high-level loop maintains prompt candidates, evaluates task traces, reflects on errors, and revises the prompt. This substantially overlaps with our RPO first-stage feedback-driven search; we explicitly present RPO as representative of this search family (Lines 283–285).

Moreover, GEPA's distinctive advantages are most relevant when several modules or prompts must be jointly optimized; it is not a second-stage local refiner comparable to GradPO. Nevertheless, we are evaluating GEPA and will share the results before the rebuttal deadline.

We will clarify the relevance of GEPA and other multi-prompt optimizers in the revised paper.

Weakness2. The evidence for optimizer-level reliability does not adequately separate optimization randomness from evaluation variability.

The five reported runs do not clearly correspond to independent repetitions of the optimization procedure. For relation extraction, Tables 4 and 5 describe variation over five test splits. For MATH-500, Appendix B.5 describes five decoding seeds applied to the same 500 problems for each selected prompt. These measurements are useful for evaluating a particular optimized prompt, but they do not establish how reliably the optimizer produces an improvement when training-example sampling, replacement generation, and prompt selection are repeated. The distinction between evaluating a particular artifact and evaluating the stochastic procedure that generates it is directly relevant here [5]. The manuscript should explicitly state the number of independent optimization runs and report the resulting distribution of gains.

Optimization randomness and evaluation variability are distinct, but independent repetitions are not a prerequisite for the validity of our controlled comparison. All refiners use identical first-stage prompts, data, selection protocol, and test evaluations. Moreover, we tune second-stage hyperparameters only on RPO prompts and transfer them unchanged to EvoPrompt-DE and ETGPO (Section 4), yet GradPO improves 18/29 cases, including 11 significant gains, more than any alternative (Lines 398–410). The matched random-span control also yields fewer improvements when gradient guidance is removed (Table 6; Lines 412–419). These results demonstrate consistency across settings rather than dependence on one favorable setting.

Repeating all optimizations with multiple seeds is computationally substantial and infeasible within the rebuttal period. We will explicitly state that each configuration was optimized once and identify run-to-run variance if the paper gets accepted.

Part (3/4)

The significance analysis also needs to respect the dependency structure of the evaluation. Appendix B.5 resamples 2,500 individual problem–seed outputs for MATH-500 and 150,000 individual query predictions for relation extraction. Outputs for the same mathematical problem share problem-specific difficulty, while queries within an episode share its support context and candidate relations. Resampling these observations as flat collections does not preserve those groupings and may understate uncertainty when generalizing to new problems or episodes. A paired cluster-aware analysis, together with uncertainty across independent optimization runs and an explicit treatment of multiple comparisons, would provide stronger support [5,6].

Thank you for this observation. Cluster bootstrapping evaluates a different uncertainty structure and is useful, but not mandatory, for our fixed-benchmark comparison. Nevertheless, we reran all significance tests using cluster bootstrapping: MATH-500 problems were resampled with all five decoding seeds, and relation-extraction episodes with all three queries.

ResultNumber of cases
Unchanged under cluster bootstrapping27/29
Changed under cluster bootstrapping2/29
Significant improvements (original)22/29
Significant improvements (clustered)20/29

The only changes are for Qwen3-4B on MATH-500: GradPO-Prob after EvoPrompt-5 and GreaTer-TG after EvoPrompt-10 are no longer significantly better within their blocks. Thus, our main conclusions remain unchanged. We will update the significance annotations accordingly.

Consequently, the reported counts of 18 improvements and 11 significant gains are useful descriptive evidence, but they do not yet establish that GradPO is a reliably better optimization procedure. This matters because consistency, rather than uniform superiority in individual settings, is the paper’s principal empirical justification.

We emphasize that our results are reliable. Tables 2–3 and Lines 398–410 provide controlled evidence across 29 settings: GradPO improves 18 prompts, achieves 11 significant gains, and is best in 10 blocks, making it the most consistent refiner. Moreover, as shown above, the cluster-aware reanalysis changes only 2/29 blocks and leaves all GradPO counts unchanged; therefore, our consistency claim remains fully supported.

Response to questions

Question1. Additional evidence needed for the main comparison. Starting from the same first-stage prompt, does GradPO retain its advantage when GReaTer/GReaTer-TG are allowed multiple updates under a matched overall refinement budget and the same validation budget? Please also apply the same protected-term constraints across methods and report actual runtime or another comparable computation measure, rather than iteration counts alone.

This is addressed under Weakness 1. With 3–5 sequential GreaTer updates matching approximately candidate-scoring calls and the same five-prompt validation budget, GradPO improves the validation score in 18/20 settings versus 9/20 and improves test performance in 11/20 versus 5/20.

Question2. Clarification and statistical reanalysis. How many independent prompt-optimization runs were performed for each configuration, as distinct from test splits and decoding seeds? Do the main conclusions, including the counts of significant improvements, remain unchanged under a paired analysis that preserves problem-level or episode-level groupings and accounts for multiple comparisons?

This is addressed under Weakness 2. Each configuration was optimized once, and cluster-aware paired testing changes only 2/29 blocks (22/29 to 20/29 significant improvements), while GradPO's 18 improvements, 11 significant gains, and 10 best results remain unchanged.

Reviewer_uTdYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage framework for automatic prompt optimization, which integrates reasoning-based global search and gradient-guided local refinement. The first stage adopts a reasoning-based optimizer to generate broad prompt improvements using task feedback, while the second stage introduces GradPO to further refine the optimized prompt by identifying important spans through target-model gradients, generating context-aware replacements, and selecting coherent edits via beam search. Compared with previous approaches that either focus on broad rewriting or token-level optimization, GradPO enables localized multi-span refinement without requiring a separate optimizer model. Extensive experiments on FS-TACRED, FS-FewRel, and MATH-500 show that the proposed framework effectively improves existing optimized prompts, with GradPO achieving consistent gains across different tasks and obtaining state-of-the-art results on FS-TACRED with Qwen3-4B.

Critical strengths

1. Effective integration of complementary prompt optimization paradigms.

The paper identifies an important limitation of existing automatic prompt optimization methods, where reasoning-based approaches are effective for broad semantic and logical revisions but often lack fine-grained control, while gradient-based methods provide local optimization signals but usually perform limited edits. By combining these two paradigms into a unified two-stage framework, the proposed approach provides a more comprehensive optimization strategy. The first stage enables global exploration of better prompt structures, while GradPO further improves the optimized prompts through targeted local refinement.

2. Well-designed gradient-guided refinement mechanism with strong empirical validation.

The proposed GradPO introduces several meaningful improvements over previous local prompt optimization methods by using target-model gradients to identify multiple high-impact spans, generating context-aware replacements with bidirectional context, and applying beam search to select coherent combinations of edits. Extensive experiments across FS-TACRED, FS-FewRel, and MATH-500 demonstrate that GradPO consistently improves prompts produced by different first-stage optimizers and achieves strong performance across both reasoning and non-reasoning tasks.

Critical weaknesses

1.The motivation and methodological novelty of combining reasoning-based optimization with gradient-guided refinement are insufficiently justified.

The paper argues that reasoning-based and gradient-based prompt optimization are complementary, but the necessity and novelty of this combination remain unclear. Recent prompt optimization methods have already explored gradient/loss-based identification of important prompt components and localized refinement (e.g., PMPO and GMPO). In this context, the proposed two-stage framework appears to mainly combine existing optimization paradigms rather than introducing a fundamentally new optimization principle. The paper lacks sufficient analysis to demonstrate why applying gradient-based refinement after reasoning-based search provides unique advantages beyond existing segment-level optimization strategies.

2.The proposed GradPO method may provide only limited lexical-level refinement rather than genuine prompt optimization.

Although GradPO introduces gradient-guided span selection and beam search, the actual optimization process mainly focuses on replacing a small number of high-gradient spans with alternative textual expressions. Many examples in the paper correspond to simple wording substitutions (e.g., replacing phrases with semantically similar alternatives), raising concerns about whether the method improves the underlying task instruction or merely performs local paraphrasing. Without a deeper analysis of the semantic changes introduced by GradPO and their relationship with model behavior, it is difficult to determine whether the proposed method achieves meaningful prompt optimization rather than surface-level editing.

3.The experimental evaluation does not sufficiently establish the superiority and robustness of GradPO.

The experimental comparisons are limited to GreaTer and LPO as second-stage refiners, while recent gradient- or segment-based prompt optimization methods are not included. Moreover, the compared methods operate under different optimization procedures and search budgets, making it unclear whether the observed improvements come from the proposed refinement mechanism or from additional search capacity. The paper also provides limited analysis of important design choices, such as the reliability of sequential multi-span editing, the dependence on the quality of first-stage prompts, and the transferability across target models. These limitations weaken the empirical evidence supporting the general effectiveness of GradPO.

Questions for authors

1. Does the sequential beam search editing remain reliable after earlier span replacements change the prompt context?

The spans and their candidates are selected before the beam-search composition, but modifying an earlier span may change the relevance or compatibility of the remaining spans. It would be useful to compare editing different numbers of spans, e.g., 1, 2, or 3 spans, or to recompute the span importance after each edit.

2. How does gradient-based span selection compare with zero-shot LLM-based span selection?

A controlled experiment that keeps candidate generation and beam search fixed would help clarify whether gradients provide a meaningful advantage over directly asking an LLM which spans should be refined.

3. Does GradPO still outperform baselines under the same computational budget?

The paper provides an analysis of the additional forward pass cost of GradPO, but the compared methods use different search budgets. An experiment of the same computational budget would help determine whether the gains come from the proposed refinement strategy itself or partly from exploring a larger search space.

4. Are the baseline comparisons sufficiently comprehensive, especially against recent segment-level prompt optimization methods?

The current evaluation mainly compares GradPO with GreaTer and LPO. However, recent methods such as PMPO, which identifies low-quality prompt segments through loss-based masking, and GMPO, which uses gradient-based segment importance followed by generated edits and loss-based selection. Comparisons with these methods would better clarify the advantages of GradPO. PMPO-https://aclanthology.org/2025.findings-emnlp.795/ GMPO-https://aclanthology.org/2026.acl-long.1089/

5. Does GradPO truly optimize prompt semantics, or mainly exploit lexical-level substitutions?

Although GradPO achieves improvements through local span replacement, many examples shown in the paper involve wording-level changes, such as replacing “description of” with “definition of”. It remains unclear whether the method can discover deeper instruction-level improvements beyond lexical refinement. Further analysis categorizing the types of edits and measuring their semantic impact would strengthen the understanding of GradPO’s optimization behavior.

Suggestions:

1.The paper contains few figures, while several tables are dense and difficult to interpret.

2.Section 3.4 mainly introduces baselines and may be better moved to the experimental section.

3.The appendix includes around 30 pages of prompt examples and templates, which seems excessive and could be substantially condensed.

5 comments

We would like to thank Reviwer uTdY for the thoughtful and constructive feedback, and we provide our responses below.

Response to weakness

  1. The motivation and methodological novelty of combining reasoning-based optimization with gradient-guided refinement are insufficiently justified.

The paper argues that reasoning-based and gradient-based prompt optimization are complementary, but the necessity and novelty of this combination remain unclear. Recent prompt optimization methods have already explored gradient/loss-based identification of important prompt components and localized refinement (e.g., PMPO and GMPO). In this context, the proposed two-stage framework appears to mainly combine existing optimization paradigms rather than introducing a fundamentally new optimization principle.

We are the first to explore two stages, and they address different issues. As stated in Lines 46-49, reasoning-based optimizers make broad semantic and logical changes, but LLMs remain sensitive to small wording choices. Further, in addition of the two-stage framework, we introduce GradPO, a second-stage optimizer that uses target-model gradients to identify and refine impactful local spans left unresolved by the first stage. Empirically, the second stage further improves already optimized prompts, and our GradPO is the most consistent second-stage refiner (Lines 398-410).

(cont.) The paper lacks sufficient analysis to demonstrate why applying gradient-based refinement after reasoning-based search provides unique advantages beyond existing segment-level optimization strategies.

To clarify, our paper already compares two established segment-level refiners: GreaTer (ICLR 2025), a gradient-based one-token method, and LPO (NAACL 2025), a span-based prompting method. Our GradPO instead combines adaptive gradient-selected spans, bidirectional replacements, and beam search using only the target model (Lines 178-211 and 232-234; Table 1).

To further clarify the papers mentioned by the reviewer: PMPO identifies segments through masking-based loss changes rather than gradients. GMPO is gradient-guided, but it appeared only very recently at ACL 2026 and uses the proprietary Claude Sonnet 4.5 Thinking as its generator and three larger judge models (20B, 14B, and 27B). In contrast, GradPO uses only one 4B target model, making it substantially simpler while remaining the most consistent refiner in our experiments.

During rebuttal, we also evaluated PMPO as a second-stage baseline on FS-TACRED and MATH-500:

MethodValidation improvementsTest improvements
GradPO18/2011/20
PMPO10/209/20

This shows that target-model gradients select useful edit locations more consistently than PMPO's masking/loss-based process, which primarily relies on a larger LLM to identify segments. We are running the FS-FewRel comparison and will report it before the rebuttal deadline. We will add, cite, and clearly distinguish both PMPO and GMPO in the revised paper.

  1. The proposed GradPO method may provide only limited lexical-level refinement rather than genuine prompt optimization.

Although GradPO introduces gradient-guided span selection and beam search, the actual optimization process mainly focuses on replacing a small number of high-gradient spans with alternative textual expressions. Many examples in the paper correspond to simple wording substitutions (e.g., replacing phrases with semantically similar alternatives), raising concerns about whether the method improves the underlying task instruction or merely performs local paraphrasing.

Localized lexical refinement is GradPO’s intended role, not evidence that it fails to optimize prompts. The first stage’s role is to improve the task instruction by making broad semantic and logical changes, while the second stage addresses the target model's sensitivity to small token choices (Lines 40–49).

The usefulness of prompt optimization is measured by the results obtained in some task, not so-called “genuine” or “meaningful” optimization (e.g., OPRO (Yang et al., 2024) and LPO (Jain and Chowdhary, 2025); see also Lines 148–156). It is trivial to get an LLM to make more detailed and verbose prompts. What matters is whether they solve the task better. In addition, we do point out that our approach does not decrease readability (Section 6), so we do not end up with random prompts that happen to work.

Part(1/4)

Part (2/4)

(cont.) Without a deeper analysis of the semantic changes introduced by GradPO and their relationship with model behavior, it is difficult to determine whether the proposed method achieves meaningful prompt optimization rather than surface-level editing.

In Table 12 and Lines 462–469, we already analyzed the different kinds of selected spans and show that only 2.4% are example- or relation-specific; the remaining spans primarily capture reusable task wording. Empirically, GradPO improves more than any alternative refiner (Lines 405–410). Thus, as explained earlier, this is GradPO's intended complementary role: it makes limited and readable local edits after broad first-stage optimization, and its usefulness is demonstrated by downstream gains.

  1. The experimental evaluation does not sufficiently establish the superiority and robustness of GradPO.

The experimental comparisons are limited to GreaTer and LPO as second-stage refiners, while recent gradient- or segment-based prompt optimization methods are not included.

This concern is addressed under Weakness 1, where we distinguish GradPO from PMPO and GMPO and report the matched PMPO comparison. Our submission already compares two recent, established segment-level refiners: GreaTer (ICLR 2025) and LPO (NAACL 2025). GMPO is both very recent and substantially more resource-intensive, using a proprietary generator and three larger judge models, whereas GradPO uses only one 4B target model. We will cite and clearly distinguish both methods.

(cont.) Moreover, the compared methods operate under different optimization procedures and search budgets, making it unclear whether the observed improvements come from the proposed refinement mechanism or from additional search capacity.

All refiners start from the same first-stage prompt and retain five prompts for the same validation procedure (Lines 367–370; Appendix B.1 and B.6). To control for search capacity, we ran experiments in this rebuttal allowing GreaTer 3–5 sequential updates to match approximately GradPO's forward calls, then evaluated the top five prompts on the same validation set across FS-TACRED and MATH-500:

MethodValidation improvementsTest improvements
GradPO18/2011/20
GreaTer9/205/20

Repeated token edits can also produce malformed text: from “Solve the following math problem,” GreaTer produced “Okay for each following math problem'' and “Okayolve the following math problem.” This shows that GradPO remains better than GreaTer under the same search budget, demonstrating the advantage of multi-span search over sequential token edits. We are running FS-FewRel and will report it before the deadline.

LPO instead uses a separate 12B/14B optimizer to reason over errors, select spans, and generate rewrites, so its generation cost is not directly comparable to target-model candidate scoring (Table 1; Appendix B.6).

(cont.) The paper also provides limited analysis of important design choices, such as the reliability of sequential multi-span editing, the dependence on the quality of first-stage prompts, and the transferability across target models. These limitations weaken the empirical evidence supporting the general effectiveness of GradPO.

We would like to clarify that all three analyses are already reported:

For multi-span reliability, every complete beam prompt is rescored after each span replacement, and the original span remains a candidate; incompatible edits can therefore be skipped rather than forced (Lines 194–211). GradPO improves 18/29 first-stage prompts (Lines 405–410), and the matched-budget GreaTer experiment above further shows that repeated one-token editing is less effective than GradPO's multi-span refinement.

Regarding first-stage quality, we use standard 5- and 10-iteration budgets, while Table 7 extends the search to 20 iterations and shows diminishing, variable gains: one GradPO step after iteration 10 still improves 7/8 settings and outperforms the best 20-iteration prompt in 5/8 (Lines 421–425).

Finally, Table 9 directly evaluates cross-model transfer; only 3/8 prompts transfer successfully because GradPO's gradients and edits are target-model-specific (Lines 1126–1130).

Part (3/4)

Response to questions

  1. Does the sequential beam search editing remain reliable after earlier span replacements change the prompt context?

The spans and their candidates are selected before the beam-search composition, but modifying an earlier span may change the relevance or compatibility of the remaining spans. It would be useful to compare editing different numbers of spans, e.g., 1, 2, or 3 spans, or to recompute the span importance after each edit.

We compute span importance once before beam search; recomputing gradients after every replacement for every beam prompt would substantially increase computation. Instead, we rescore each resulting complete prompt and retain the original span text as a candidate, so a selected span can remain unchanged if its replacements no longer fit (Lines 194–211 and 807–809). GradPO therefore edits only 2.76 spans on average, rather than all selected spans (Table 11). We will clarify this trade-off.

  1. How does gradient-based span selection compare with zero-shot LLM-based span selection?

A controlled experiment that keeps candidate generation and beam search fixed would help clarify whether gradients provide a meaningful advantage over directly asking an LLM which spans should be refined.

Our controlled random-span experiment keeps candidate generation and beam search unchanged; gradient-selected spans improve 18/29 test cases, compared with 11/29 for random spans (Table 6; Lines 412–419). LPO provides a practical LLM-guided alternative, where an LLM selects edit spans, but improves only 9/29 cases (Lines 405–410). Although LPO is not a selector-only control, together these results show that gradient-guided selection is more consistent than random or LLM-guided local editing.

  1. Does GradPO still outperform baselines under the same computational budget?

The paper provides an analysis of the additional forward pass cost of GradPO, but the compared methods use different search budgets. An experiment of the same computational budget would help determine whether the gains come from the proposed refinement strategy itself or partly from exploring a larger search space.

This is addressed under Weakness 3. Under the matched search and validation budgets, GradPO improves the validation score in 18/20 settings versus 9/20 for GreaTer and improves test performance in 11/20 versus 5/20. The examples there also show that repeated one-token edits can produce malformed prompts. We will add these results and the pending FS-FewRel comparison to the revised paper.

  1. Are the baseline comparisons sufficiently comprehensive, especially against recent segment-level prompt optimization methods?

The current evaluation mainly compares GradPO with GreaTer and LPO. However, recent methods such as PMPO, which identifies low-quality prompt segments through loss-based masking, and GMPO, which uses gradient-based segment importance followed by generated edits and loss-based selection. Comparisons with these methods would better clarify the advantages of GradPO. PMPO-https://aclanthology.org/2025.findings-emnlp.795/ GMPO-https://aclanthology.org/2026.acl-long.1089/

As discussed under Weakness 1, our submission already compares the recent segment-level refiners GreaTer (ICLR 2025) and LPO (NAACL 2025). Our new PMPO comparison also shows that GradPO improves first-stage prompts more consistently on FS-TACRED and MATH-500 (11/20 versus 9/20 test improvements). GMPO is very recent (ACL 2026) and substantially more resource-intensive, using Claude Sonnet 4.5 Thinking as its generator and three larger judge models. Even if GMPO reports stronger results, GradPO remains an empirically effective and substantially simpler alternative, using only the 4B target model. We will add and clearly distinguish both PMPO and GMPO in the revised paper.

  1. Does GradPO truly optimize prompt semantics, or mainly exploit lexical-level substitutions?

Although GradPO achieves improvements through local span replacement, many examples shown in the paper involve wording-level changes, such as replacing “description of” with “definition of”. It remains unclear whether the method can discover deeper instruction-level improvements beyond lexical refinement. Further analysis categorizing the types of edits and measuring their semantic impact would strengthen the understanding of GradPO’s optimization behavior.

This concern is addressed under Weakness 2. GradPO is intentionally a local refiner: the first stage makes broad semantic and logical changes, while GradPO targets model-sensitive wording left unresolved by the first stage (Lines 40–49). Our results show that these local edits improve already optimized prompts (Lines 405–410) while largely preserving prompt readability (Section 6; Lines 452–460). Ultimately, prompt optimization should be judged by downstream task improvement, since deeper or broader semantic rewrites are not necessarily more useful.

Comments on the paper

1 comment

Dear reviewers,

The authors have submitted their rebuttals. Please take a look and interact with the authors if necessary.

Thanks a lot.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.