acceptodds
Under review as a conference paper at ICLR 2027

What Should the Reflector See? An Empirical Study of Evidence in Reflective Prompt Optimization

Abstract

Reflective prompt optimization revises instructions using examples of a model’s behavior, but which evidence the reflector should receive remains unclear. We compare nine reflection strategies within a single-parent Pareto-guided search, varying evidence composition, visibility of examples, candidate selection and domain-knowledge policy. Using Qwen3.5-9B as both task model and reflector, we complete 45 optimization runs on five datasets and report test accuracy averaged over three final evaluations. No strategy dominates across tasks. Failure-only evidence produces the largest average improvement, driven by a 33.7-point gain on LiveBench-Math, whereas balanced evidence and validated blind generation share the best mean accuracy rank. Balanced evidence is not uniformly better than failure-only evidence, and explicit knowledge retention helps multi-hop question answering but hurts some other tasks in these runs. Empirical selection with natural evidence outperforms an LLM judge on average, although their computational costs differ. The results highlight task dependence and the need to distinguish calibration improvement from held-out performance, with repeated optimization runs still needed for stronger conclusions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
prompt evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dr3AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies automated prompt optimization. Specifically, the authors analyze what information should be provided to the reflector. Using Qwen3.5-9B, they evaluate different strategies for selecting the content shown to the reflector.

Critical strengths

(1) Authors tackle automated prompt optimization problem, which is a timely problem.

Critical weaknesses

(W1) It is difficult to draw general conclusions from the authors’ experiments because they evaluate only one (task model, reflector, LLM judge) triple, using Qwen3.5-9B for all three roles. This is particularly important because the conclusions may differ across model families and sizes. Since the paper’s contribution is primarily empirical, with none algorithmic novelty, I consider the lack of model diversity an important experimental limitation.

(W2) I also find it difficult to draw strong conclusions from the reported experiments on this one triple considered. For example, the authors state that Failures-only achieves the largest average gain, but this result is dominated by a single benchmark:

DatasetFailures-only gain (pp)
LiveBench-Math+33.7
OlympiadBench+7.6
GPQA−3.1
HotpotQA−0.7
IFBench+2.3
Mean+8.0

The authors acknowledge this in the text, but it leaves little basis for a general conclusion.

Therefore, I do not fully understand the benefit of the authors’ analysis. Their main conclusion is essentially that what the reflector should see depends on the benchmark, which is not particularly surprising. Although the authors do not investigate this, the result also likely depends on the model family and model size.

(W3) As the authors explain in Table 2, It.=0 means that the original seed prompt was retained. However, the reported results for these cases are:

HotpotQA It.=0Accuracy
Seed baseline37.6
Failures-only36.9
Balanced-mix36.9
No-examples36.7
No-examples+Val37.6
LLM-judge36.2
No-knowledge37.3

Thus, the same seed prompt obtains results ranging from 36.2 to 37.6. This variation is severe relative to several of the reported differences between strategies. In the automatic prompt optimization literature, task-model evaluation is often performed with greedy decoding to reduce this source of variance, leaving the variation in the prompts produced by the optimizer.

(W5) The authors also do not describe their evaluation protocol sufficiently clearly. In particular, it is unclear how many times each complete strategy is independently rerun to obtain a new optimized prompt. Although this is not stated directly, a reader can infer that paper uses only one optimization run per setting: Section 4.2 states that “The completed grid has (5×9=45(5 \times 9 = 45 runs,” while Section 4.4 states that “Every final prompt and its corresponding seed prompt are evaluated three times on the same held-out test examples.”. This is also a protocole that is not met in most of the automated prompt optimization literature, where authors for each seed optimize for new prompt.

(W6) Since the authors report results averaged across three final evaluation passes, they should also provide the standard deviations.

Questions for authors

see weaknesses

Reviewer_7WMFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies what evidence should be given to a reflector during prompt optimization. It compares nine strategies on five datasets using Qwen3.5-9B. Failure examples give the largest average gain, while other strategies rank better across tasks. The paper also separates final performance, local improvement, and transfer to test data.

Critical strengths

The research question is clear and useful. The study compares several reasonable evidence choices under one search framework. I also like the distinction between improving a sampled parent and improving the final prompt. The authors describe important limits carefully and provide the reflection templates and evolved prompts for checking results

Critical weaknesses

The main results use one optimization run per setting. Three final evaluations cannot show search stability. Also, Failures-only has an easier local improvement gate because its parent examples are all wrong. This makes evidence effects difficult to separate from selection effects. A shared validation batch would make comparisons more convincing.

Questions for authors

How much of the LiveBench gain comes from fixing answer formatting rather than solving more problems? Appendix A.3.1 suggests this may matter. Can you repeat searches with several random seeds and select all candidates on a common validation batch? Would the ranking remain similar under the same total model-call budget?

Reviewer_RGrXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies what evidence the reflector should see in reflective prompt optimization, for example only failures, a mix of failures and successes, or no examples at all, along with how new prompts are selected and whether domain knowledge is kept. It compares nine such strategies within a single GEPA-style search, using one small model on five datasets. The authors find that no strategy wins everywhere, that improving the current prompt locally does not reliably produce a better final prompt, and that calibration gains can overstate held-out gains, so they recommend judging strategies on several measures and validating them on the target task.

Critical strengths
  1. The setup is focused and open. All strategies share the same search, seed prompt, splits and number of iterations, and the paper is clear about where they differ in gating and cost. It also publishes all 45 final prompts, so readers can see what each strategy actually learned. That is exactly what made the second weakness below visible.

  2. The paper makes good points about how to judge prompt optimizers. Winning locally doesn't mean a better final prompt: the no-examples strategy improves its parent most often but ends with the worst accuracy rank. Calibration gains can also overstate held-out gains on some tasks. Reporting held-out gains, and progress over the best prompt so far, is good advice.

Critical weaknesses
  1. There is only one optimization run per strategy and dataset, so the comparisons can't carry the conclusions.
  • The three test runs just re-run the same final prompt. They measure decoding noise, not the much bigger randomness of the search itself (reflection at temperature 1.0, random training batches, random parent picks).
  • Even the decoding noise is as big as many of the differences. On HotpotQA, six of the nine strategies never move off the seed prompt, but their scores still spread by more than a point.
  • So the pairwise comparisons, the rank summaries and the takeaways may mostly show which run got lucky. One 9B model plays every role with thinking turned off, so it is also unclear whether any ranking would hold with a stronger reflector, which is the usual GEPA setup.
  1. The biggest effect looks like an answer-format trick, not better reasoning advice.
  • Most of the headline gain for Failures-only comes from LiveBench-Math. In the appendix, the winning prompt tells the model to ignore the question's request to repeat the answer letter five times and to put one letter in the box. The runner-up even quotes the scorer's feedback: "answer was 'EEEEE' but correct is 'E'".
  • LiveBench's competition questions ask for the letter five times, but the paper's scorer only accepts one letter. Prompts that follow the question's own instruction score well below the top two.
  • Many math prompts also focus on not running out of tokens, which the scorer counts as a failure.
  • So the math results may mostly show which strategy stumbled onto the scorer's rules, not which gives better reasoning advice. That goes against the paper's own explanation. The paper doesn't report format or truncation failure rates that could tell the two apart.
Questions for authors
  1. How many LiveBench-Math test items have the "repeat the letter" instruction? What are the results with LiveBench's official grader, or with a lenient answer extractor used for every strategy?

  2. For each strategy and dataset, what share of errors are format/parse failures or truncations, before and after optimization?

  3. Can you run several optimization seeds per cell (cheap with a 9B model) and report the spread and paired tests for the main comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.