- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
UPA optimizes prompts without ground-truth labels: an MCTS-style tree is grown from judge-LLM child-vs-parent comparisons, a five-point Likert comparison is debiased by swapping order, and a two-stage selection picks the final prompt (path-wise Beta posterior + LCB top-K; global round-robin + BTL/MM). It averages 69.3% on five closed-ended benchmarks vs 67.5% for PDO and 66.3% for SPO.
Critical strengths
The method is specified well enough to reproduce: both algorithms, all hyperparameters, the judge and optimizer prompts, and code are provided. The paper reports a same-budget comparison of search organisations (Table 12: breadth-first 66.8, population 64.7, pool-based 65.5, independent-path 66.8 vs UPA 69.3), a control that prompt-optimization papers usually omit. Empirical coverage is also non-trivial: five closed-ended benchmarks, three execution LLMs plus Qwen3.5-2B, three MT-Bench sub-tasks, and a per-phase cost breakdown (Table 7), with Stage II costing only 4.3% of the optimisation budget.
Critical weaknesses
-
The claim is not supported. The gain over the strongest GT-free baseline is +1.8 average (Table 1, ≈1.5 SE); cross-model margins are +0.3–0.4 (Table 9, with all eleven methods within 89.6–91.4 on GPT-5); open-ended win rates against SPO/PDO are mostly 50.8–58.3%, with ~60 judgments per cell (all 45 values are 1/120 multiples), i.e. within noise. No confidence interval or significance test appears anywhere.
-
The closest neighbour is missing. DEEVO (Nair et al., 2025), one of the paper's three motivating GT-free methods and a pairwise-debate + Elo selector, appears in no table; overlapping MT-Bench/BBH protocols make a matched-budget run feasible. Its reported MT-Bench win rates over SPO (66.7–85%) are also well above UPA's (50.8–70%), though under a different protocol.
-
Judge validity is untested. Nothing checks whether judge preferences track true quality although the closed-ended tasks allow it; the only reported correlation (0.618) is Stage-I vs Stage-II, not against truth. The optimization judge is the executor itself, and App. C.9 shows the judge preferring a response whose stated order contradicts the option it selects because it is "clearer".
-
Reporting. Most baseline numbers are imported from SPO's paper with no budget counts; the "40% of PDO cost" and "SPO at 65.2%" claims are untabulated; the open-ended protocol (number of judgments, tie handling, order randomisation) and the split used for Fig. 3 are unstated; the Venue column is cosmetic; Q(v) is a back-propagated local improvement, not a node-quality estimate.
Questions for authors
- Provide confidence intervals and paired tests for Tables 1–5, 9 and 12 — are +1.8 and +0.3–0.4 distinguishable from zero?
- Report judge-vs-GT agreement (per-node Spearman, P(selected = GT-best)).
- Run UPA under DEEVO's protocol and report the comparison, or explain why it is impossible.
9 comments
[2/5] Response to Reviewer jseh
W3 and Q2: Judge accuracy and model roles
1. Agreement with ground truth
We measure judge agreement with ground truth (GT) on pairs where exactly one answer is correct. If both answers are correct or both are wrong, their reasoning and overall quality can still differ. Their final-answer labels alone do not determine which response is better overall or whether they should tie. The judge favors the correct response in 3,862 of 4,468 eligible judgments (86.44%). Across child nodes, we correlate the local judge reward R(v) with the change in GT accuracy from parent to child. Both use the same five comparison questions for that node. R(v) averages the judge's scores after combining both presentation orders, as described in Section 2.2 and Eq. (7). The Spearman correlation is 0.813. This checks whether higher judge scores accompany larger accuracy gains. The previously reported 0.618 compares Stage I and Stage II rankings. We have added both judge-versus-GT analyses to Appendix C.5 of the revised manuscript.
2. Separate execution and judging roles
The executor generates answers using a candidate prompt. The optimization judge compares two answers using the dedicated prompt in Appendix B.3. Our default configuration uses GPT-4o-mini for both roles, with temperatures of 0.0 and 0.3, respectively. The roles use different prompts and can be assigned different models. Appendix C.8 (Table 11) of the original manuscript already reported the following results with GPT-4o-mini fixed as the executor and GPT-4o fixed as the optimizer:
| Optimization judge | Executor | Avg. accuracy (%) |
|---|---|---|
| GPT-4o-mini | GPT-4o-mini | 69.3 |
| GPT-4o | GPT-4o-mini | 69.4 |
| Claude-4.5-Sonnet | GPT-4o-mini | 69.3 |
3. The example in Appendix C.9 of the original manuscript
In Appendix C.9 of the original manuscript, the GT answer is option C. Response A selects C, while Response B states that none of the options matches its reasoning and then selects option A, which is wrong. Although A's intermediate ordering is inconsistent with its selected option, the judge's A_BETTER decision favors the response with the correct final answer.
W4: Optimization costs and reporting details
1. Optimization costs and performance
The baseline costs reported in Appendix C.4 (Figure 4) of the original manuscript, except PDO, were taken from SPO. We have now computed the costs from the measured input and output token usage of our own runs. The table below gives these costs and the accuracies from the three-run Table 1 comparison, together with the additional Scaled SPO result.
| Metric | APE | OPRO | PromptAgent | PromptBreeder | TextGrad | SPO | Scaled SPO | DEEVO | PDO | UPA |
|---|---|---|---|---|---|---|---|---|---|---|
| Optimization cost (USD) | 9.32 | 4.47 | 3.01 | 5.25 | 15.10 | 0.22 | 1.42 | 4.58 | 3.55 | 1.41 |
| Avg. accuracy (%) | 65.0 | 66.2 | 66.4 | 64.3 | 64.4 | 65.5 | 65.2 | 66.8 | 67.5 | 69.3 |
UPA's optimization cost is USD 1.41, compared with USD 3.55 for PDO, or approximately 40% of PDO's cost. At a comparable budget of USD 1.42, Scaled SPO reaches 65.2% average accuracy, compared with UPA's 69.3%. These costs apply to prompt optimization. The optimized prompts are used directly at test time.
2. Evaluation details and search statistics
- Open-ended evaluation. Appendix C.3 now specifies the judgment counts, tie handling and evaluation in both presentation orders.
- Figure 3 split. Figure 3 reports accuracy on the same final test sets used in Tables 1, 2, 4 and 5. The optimization and test partitions are described in Appendix C.1.
- Q(v) and Stage I. The review's description of Q(v) matches its definition in Section 2.3 and Eqs. (4) and (8). It is the mean of local comparison rewards backpropagated to the node and is used to guide search. Stage I Bayesian filtering addresses the subsequent prompt-selection problem: under the BTL model, it combines edge comparisons along each path to estimate prompt quality relative to the root, accounts for uncertainty, and filters candidates for Stage II. The selection ablation shows that Max Q(v) gives 65.1% average accuracy, Stage I alone gives 68.3%, and the complete procedure gives 69.3%.
- Venue column. We have removed the Venue column from Table 1.
[3/5] Response to Reviewer jseh
Detailed results
Table numbers below follow the original manuscript, except for the explicitly marked new table (revised Table 9). Accuracy and win rates are percentages. Results are mean ± sample std across three independent runs, with 30 runs for the new PDO comparison. Avg. is calculated within each run. CI denotes a 95% confidence interval of the mean, and Ref. identifies UPA as the reference for paired accuracy comparisons. Bold accuracy values indicate the highest means, and bold p-values are below 0.05.
Table 1. Reproduced closed-ended benchmark results
| Method | GPQA | MATH | LIAR | WSC | BBH | Avg. | 95% CI of Avg. | p for Avg. |
|---|---|---|---|---|---|---|---|---|
| IO | 40.9 ± 0.5 | 40.7 ± 0.6 | 65.4 ± 1.5 | 76.2 ± 1.0 | 89.7 ± 0.8 | 62.6 ± 0.4 | [61.5, 63.7] | 0.005 |
| CoT | 41.7 ± 0.8 | 46.9 ± 0.4 | 64.0 ± 1.8 | 78.9 ± 2.3 | 88.5 ± 1.0 | 64.0 ± 0.5 | [62.9, 65.1] | 0.006 |
| RaR | 38.7 ± 0.3 | 41.5 ± 0.5 | 51.1 ± 0.6 | 81.5 ± 2.0 | 97.2 ± 1.3 | 62.0 ± 0.5 | [60.8, 63.3] | < 0.001 |
| APE | 42.6 ± 0.3 | 47.2 ± 1.7 | 64.2 ± 0.8 | 78.5 ± 1.7 | 92.3 ± 0.8 | 65.0 ± 0.7 | [63.1, 66.8] | 0.023 |
| OPRO | 41.6 ± 0.8 | 46.0 ± 1.4 | 65.6 ± 2.5 | 81.8 ± 1.7 | 96.2 ± 1.0 | 66.2 ± 0.5 | [64.9, 67.5] | 0.016 |
| PromptAgent | 44.2 ± 0.8 | 44.9 ± 1.7 | 67.2 ± 0.6 | 81.1 ± 1.7 | 94.7 ± 0.8 | 66.4 ± 0.6 | [65.0, 67.9] | 0.010 |
| PromptBreeder | 38.6 ± 1.5 | 47.4 ± 1.1 | 61.5 ± 1.2 | 77.1 ± 1.7 | 97.0 ± 2.6 | 64.3 ± 0.7 | [62.4, 66.2] | 0.009 |
| TextGrad | 39.9 ± 0.5 | 44.8 ± 2.6 | 66.1 ± 1.0 | 77.8 ± 2.0 | 93.3 ± 2.5 | 64.4 ± 0.4 | [63.5, 65.3] | < 0.001 |
| SPO | 43.6 ± 1.5 | 42.2 ± 1.8 | 65.9 ± 1.4 | 80.0 ± 1.8 | 95.7 ± 1.8 | 65.5 ± 0.9 | [63.3, 67.7] | 0.007 |
| DEEVO | 43.7 ± 0.8 | 45.6 ± 1.0 | 66.7 ± 1.2 | 80.7 ± 0.7 | 97.3 ± 1.8 | 66.8 ± 0.6 | [65.4, 68.2] | 0.001 |
| PDO | 43.6 ± 1.3 | 47.7 ± 1.8 | 67.9 ± 0.4 | 82.0 ± 0.7 | 96.2 ± 0.8 | 67.5 ± 0.2 | [66.8, 68.1] | 0.014 |
| UPA | 45.5 ± 1.1 | 52.1 ± 1.2 | 68.2 ± 0.3 | 82.7 ± 0.7 | 98.0 ± 1.0 | 69.3 ± 0.4 | [68.2, 70.4] | Ref. |
New table. PDO vs UPA over 30 independent runs (revised Table 9)
| Method | GPQA | MATH | LIAR | WSC | BBH | Avg. | 95% CI of Avg. | p for Avg. |
|---|---|---|---|---|---|---|---|---|
| PDO | 43.4 ± 1.3 | 47.6 ± 1.8 | 67.4 ± 0.7 | 81.4 ± 1.1 | 96.2 ± 1.0 | 67.2 ± 0.6 | [67.0, 67.4] | < 0.001 |
| UPA | 45.5 ± 1.3 | 52.4 ± 0.9 | 68.2 ± 0.8 | 82.9 ± 1.1 | 98.3 ± 1.1 | 69.5 ± 0.5 | [69.3, 69.6] | Ref. |
Table 2. Transfer to other execution models
| Executor | Method | GPQA | MATH | LIAR | WSC | BBH | Avg. | 95% CI of Avg. | p for Avg. |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5 | SPO | 82.5 ± 2.3 | 94.9 ± 1.4 | 78.0 ± 0.9 | 98.2 ± 0.8 | 98.5 ± 0.5 | 90.4 ± 0.7 | [88.6, 92.2] | 0.021 |
| GPT-5 | DEEVO | 83.5 ± 1.9 | 93.4 ± 0.4 | 78.2 ± 1.0 | 97.1 ± 1.0 | 99.3 ± 0.8 | 90.3 ± 0.4 | [89.2, 91.4] | 0.038 |
| GPT-5 | PDO | 83.3 ± 2.2 | 95.2 ± 2.2 | 78.3 ± 0.6 | 97.6 ± 1.0 | 99.0 ± 0.5 | 90.7 ± 0.6 | [89.2, 92.1] | 0.015 |
| GPT-5 | UPA | 84.2 ± 2.1 | 95.7 ± 1.4 | 78.8 ± 0.8 | 98.5 ± 0.4 | 100.0 ± 0.0 | 91.4 ± 0.6 | [89.9, 93.0] | Ref. |
| Claude-4.5-Sonnet | SPO | 73.7 ± 1.6 | 84.7 ± 1.7 | 74.1 ± 0.7 | 97.3 ± 0.7 | 99.8 ± 0.3 | 85.9 ± 0.3 | [85.2, 86.6] | 0.037 |
| Claude-4.5-Sonnet | DEEVO | 75.1 ± 0.8 | 84.9 ± 0.6 | 73.7 ± 0.6 | 96.0 ± 0.7 | 98.3 ± 0.8 | 85.6 ± 0.4 | [84.6, 86.6] | 0.017 |
| Claude-4.5-Sonnet | PDO | 74.7 ± 0.6 | 85.8 ± 1.0 | 74.2 ± 2.6 | 97.1 ± 1.0 | 99.5 ± 0.5 | 86.3 ± 0.6 | [84.8, 87.7] | 0.060 |
| Claude-4.5-Sonnet | UPA | 75.8 ± 1.1 | 86.6 ± 0.2 | 74.7 ± 0.8 | 98.0 ± 0.7 | 100.0 ± 0.0 | 87.0 ± 0.3 | [86.3, 87.7] | Ref. |
| DeepSeek-V3.2 | SPO | 73.6 ± 2.4 | 86.3 ± 1.8 | 68.2 ± 0.1 | 93.1 ± 0.3 | 99.3 ± 0.3 | 84.1 ± 0.7 | [82.4, 85.9] | 0.057 |
| DeepSeek-V3.2 | DEEVO | 74.4 ± 1.1 | 89.9 ± 1.7 | 67.8 ± 1.1 | 92.7 ± 0.7 | 97.7 ± 0.8 | 84.5 ± 0.5 | [83.3, 85.7] | < 0.001 |
| DeepSeek-V3.2 | PDO | 75.1 ± 1.7 | 87.8 ± 1.5 | 68.3 ± 0.9 | 93.3 ± 0.7 | 99.3 ± 0.8 | 84.8 ± 0.8 | [82.9, 86.6] | 0.031 |
| DeepSeek-V3.2 | UPA | 78.3 ± 1.0 | 93.1 ± 0.6 | 68.6 ± 0.5 | 93.8 ± 0.4 | 99.7 ± 0.3 | 86.7 ± 0.5 | [85.4, 88.0] | Ref. |
[1/5] Response to Reviewer jseh
We thank the reviewer for the detailed feedback. We appreciate your positive comments on UPA's reproducibility, the comparison of search strategies under the same budget, and the experiments across tasks and models. Following your suggestions, we have reproduced the baselines, compared UPA with DEEVO, and evaluated the judge's accuracy. We have added these results, along with statistical analyses and evaluation details, to the revised manuscript.
W1, W2, Q1 and Q3: Statistical results and DEEVO comparison
1. Repeated experiments and statistical results
2. MT-Bench evaluation and additional runs
3. Direct comparison with DEEVO
The detailed results are provided in Parts 3–5.