- Rating
- 3
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RBEPO optimizes prompts through failure-mode diagnosis, targeted branching, merging of effective repairs, and reuse of editing experience. It also introduces fragility-guided racing, which prioritizes examples historically prone to regression to eliminate unfavorable candidates early. Across four public tasks and two target models, the reported results outperform the compared baselines, while the racing strategy reduces sample evaluations with performance close to full evaluation.
Critical strengths
The paper addresses a worthwhile problem with a coherent design. Connecting failure modes, concrete edits, and observed outcomes is a promising way to guide prompt optimization. The consistently positive results across eight model–task combinations are encouraging. The work also considers both candidate quality and evaluation efficiency, and the comparison with random racing provides useful evidence for exploiting historical regression information.
Critical weaknesses
-
The individual contributions of the mechanisms are not sufficiently isolated. The cumulative ablations show that simplifying the system reduces performance, but do not clearly separate component effects and interactions. Removing branching also reduces candidate generation and search opportunities, making it difficult to attribute the gain specifically to failure-mode organization. Similarly, the experience-reuse analysis does not establish the additional value of structured mechanism-level experience over simpler historical feedback.
-
The trade-off between early elimination and overall improvement remains unclear. Prioritizing regression-prone examples can expose harmful edits quickly, but may also discard candidates that lose locally while improving overall. The reported results demonstrate a promising performance–evaluation trade-off, yet provide limited analysis of eliminated candidates and their influence on subsequent search. It is therefore unclear how much efficiency comes from identifying poor candidates versus restricting exploration more conservatively.
Questions for authors
- Can representative optimization trajectories show how failure modes and accumulated experience change subsequent edits, and what they contribute beyond ordinary feedback summaries?
- Can a sample of eliminated candidates be fully evaluated to assess whether useful overall improvements are missed and how such eliminations affect the final search outcome?
3 comments
Response to Reviewer Jwpf (Part 2 of 2)
Question 2: Follow-Up Evaluation of Eliminated Candidates and the Final Search Outcome
An early-eliminated candidate is counted as a “mistaken elimination” if its overall score, after completing evaluation on the full DEV set, exceeds that of its actual parent prompt. For merged candidates, the reference is the strongest actual parent prompt.
Of the 86 eliminated candidates in this follow-up evaluation, 2 meet the above criterion for mistaken elimination, accounting for 2.33%. In the full optimization comparison, the four-task average TEST scores without racing and with fragility-guided racing are 68.27 and 67.61, respectively.
Table R3: Final TEST scores and follow-up evaluations of candidates eliminated by fragility-guided racing (qwen3.8-flash, scores on a 0–100 scale).
| Task | TEST without racing | TEST with fragility-guided racing | Eliminated candidates | Mistakenly eliminated candidates | Mistaken elimination percentage |
|---|---|---|---|---|---|
| finqa | 75.60 | 74.00 | 24 | 1 | 4.17% |
| hupd | 49.00 | 48.00 | 21 | 0 | 0.00% |
| ecthr_a | 77.49 | 78.25 | 22 | 0 | 0.00% |
| hotpot_qa | 71.00 | 70.20 | 19 | 1 | 5.26% |
| Average / Total | 68.27 | 67.61 | 86 | 2 | 2.33% |
Candidate Counts and Branching Ablation
Thank you for suggesting comparisons with the same number of candidates. All settings in the original ablation use a total of 30 proposals. Settings with branching run for 10 rounds with 3 candidates per round; settings without branching run for 30 rounds with 1 candidate per round. We add the round and proposal counts to the original ablation table below.
Table R4: The original cumulative ablation, with round and proposal counts added (qwen3.8-flash, TEST, 0–100 scale). Each row removes the listed mechanism in addition to those removed in the preceding row.
| Variant | finqa | hupd | ecthr_a | hotpot_qa | Avg | Rounds | Proposals |
|---|---|---|---|---|---|---|---|
| Full | 74.00 | 48.00 | 78.25 | 70.20 | 67.61 | 10 | 30 |
| − experience reuse | 73.40 | 46.60 | 77.02 | 69.20 | 66.56 | 10 | 30 |
| − branching and merging | 72.80 | 44.60 | 76.15 | 67.60 | 65.29 | 30 | 30 |
| − structured diagnosis | 69.60 | 43.00 | 75.37 | 65.40 | 63.34 | 30 | 30 |
In addition, with both diagnosis and experience disabled, we compare two settings: 3 candidates per round for 10 rounds, and 1 candidate per round for 30 rounds. With the same total of 30 proposals, the two settings achieve similar four-task average performance.
Table R5: Supplementary candidate-generation comparison (qwen3.8-flash, TEST, 0–100 scale).
| Setting | finqa | hupd | ecthr_a | hotpot_qa | Avg | Rounds | Proposals |
|---|---|---|---|---|---|---|---|
| Full | 74.00 | 48.00 | 78.25 | 70.20 | 67.61 | 10 | 30 |
| − experience reuse − diagnosis (3 candidates per round) | 71.20 | 41.20 | 75.00 | 67.20 | 63.65 | 10 | 30 |
| − experience reuse − diagnosis − branching and merging (1 candidate per round) | 69.60 | 43.00 | 75.37 | 65.40 | 63.34 | 30 | 30 |
Summary
We have added candidate comparisons, an experience-reuse example, follow-up evaluations of eliminated candidates, and a performance comparison with and without racing. Thank you for your suggestions. We hope these clarifications and supplementary results address your main concerns, and respectfully ask you to reconsider your assessment and rating of the paper in light of this information. If you have any remaining questions, please let us know; we would be happy to provide further clarification.
Thank you for additional experiments, clarification and cases study. The response addresses several of my concerns (W1, W2, Q2), and the case study illustrates how feedback from earlier edits can guide later ones. However, it remains unclear whether a simple summary of the same edit history would perform as well as proposed structured approach. So I will maintain my score.
Response to Reviewer Jwpf (Part 1 of 2)
Thank you for recognizing the value of our research question and our design linking failure modes, concrete edits, and observed outcomes. Below, we address your questions about mechanism contributions and the final search outcome through optimization trajectories, follow-up evaluations of eliminated candidates, and candidate comparisons.
Question 1: Mechanism-Level Experience for Guiding Subsequent Edits
In RBEPO, diagnosis and branching-based optimization make explicit the problem targeted by each rewrite, the concrete edits, and their gains and regressions. We further organize this information into problem mechanisms, rewriting strategies, improvements, and negative effects, distilling structured mechanism-level experience for subsequent edits. Ordinary feedback summaries can also contain this information; our design emphasizes explicitly linking and organizing it, so that experience distillation follows naturally from branching-based rewriting and validation and provides a basis for subsequent optimization.
Example: FinQA (qwen3.8-27b). This example shows how experience from an unselected rewrite is distilled and used to guide subsequent optimization. The parent–child and experience-reuse relationships among the versions are shown below:
Here, the unselected v7 still provides reusable experience. Table R1 presents its target problem, rewriting strategy, validation results, and analysis of regression causes.
Table R1: Experience distilled from the v7 rewrite (FinQA, qwen3.8-27b, the same 416 DEV examples for v2 and v7). Repairs and regressions are measured relative to the parent prompt v2.
In the next round, v5 serves as the parent prompt, while v7 supplies the experience. Building on the parent's existing convention for percentage outputs, v8 adds rules for decomposing computations and referencing results, states that subsequent steps can use these results, and includes correct multi-step examples.
On the full DEV set, v8 repairs 82 examples and introduces 2 regressions relative to v5, raising the score from 57.2 to 73.2 and eliminating nested programs.
Table R2: Results of the subsequent edit (FinQA, qwen3.8-27b, the full set of 500 DEV examples). Repairs and regressions are measured relative to the parent prompt v5.