- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies automated prompt optimization. Specifically, the authors analyze what information should be provided to the reflector. Using Qwen3.5-9B, they evaluate different strategies for selecting the content shown to the reflector.
Critical strengths
(1) Authors tackle automated prompt optimization problem, which is a timely problem.
Critical weaknesses
(W1) It is difficult to draw general conclusions from the authors’ experiments because they evaluate only one (task model, reflector, LLM judge) triple, using Qwen3.5-9B for all three roles. This is particularly important because the conclusions may differ across model families and sizes. Since the paper’s contribution is primarily empirical, with none algorithmic novelty, I consider the lack of model diversity an important experimental limitation.
(W2) I also find it difficult to draw strong conclusions from the reported experiments on this one triple considered. For example, the authors state that Failures-only achieves the largest average gain, but this result is dominated by a single benchmark:
| Dataset | Failures-only gain (pp) |
|---|---|
| LiveBench-Math | +33.7 |
| OlympiadBench | +7.6 |
| GPQA | −3.1 |
| HotpotQA | −0.7 |
| IFBench | +2.3 |
| Mean | +8.0 |
The authors acknowledge this in the text, but it leaves little basis for a general conclusion.
Therefore, I do not fully understand the benefit of the authors’ analysis. Their main conclusion is essentially that what the reflector should see depends on the benchmark, which is not particularly surprising. Although the authors do not investigate this, the result also likely depends on the model family and model size.
(W3) As the authors explain in Table 2, It.=0 means that the original seed prompt was retained. However, the reported results for these cases are:
HotpotQA It.=0 | Accuracy |
|---|---|
| Seed baseline | 37.6 |
| Failures-only | 36.9 |
| Balanced-mix | 36.9 |
| No-examples | 36.7 |
| No-examples+Val | 37.6 |
| LLM-judge | 36.2 |
| No-knowledge | 37.3 |
Thus, the same seed prompt obtains results ranging from 36.2 to 37.6. This variation is severe relative to several of the reported differences between strategies. In the automatic prompt optimization literature, task-model evaluation is often performed with greedy decoding to reduce this source of variance, leaving the variation in the prompts produced by the optimizer.
(W5) The authors also do not describe their evaluation protocol sufficiently clearly. In particular, it is unclear how many times each complete strategy is independently rerun to obtain a new optimized prompt. Although this is not stated directly, a reader can infer that paper uses only one optimization run per setting: Section 4.2 states that “The completed grid has runs,” while Section 4.4 states that “Every final prompt and its corresponding seed prompt are evaluated three times on the same held-out test examples.”. This is also a protocole that is not met in most of the automated prompt optimization literature, where authors for each seed optimize for new prompt.
(W6) Since the authors report results averaged across three final evaluation passes, they should also provide the standard deviations.
Questions for authors
see weaknesses