- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper studies reflective prompt optimizers like GEPA, which suffers from prompt saturation - where the system keeps adding instruction texts to the prompt even when the score has stopped improving. RB-Prompt tackles this by watching each search branch, and flags when a branch when over the last k accepted rewrites the score gain is below a threshold but the instruction grew above another threshold. When a branch is flagged, RB-PRompt that rolls back to the branch's best prompt and ask a reflection model to remove/combine rules, and then applies a relative length penalty to later candidates. Parents are picked with a TPE density ratio over past rewrite gains. Expts use HotPotQA, GSM8K and another proprietary PDF extraction task, and used Claude Sonnet 4.6, GPT-4o and Qwen-2.5-VL-72b, with 16 runs per settings. Their method has the best or tied mean in all nine settings (Tab. 2). It gains 7.4-10.0 pts over GEPA on HPQA, 6.0-10.0 pts on extraction, but only 0.6-1.2 pts on GSM8K. They also show that repair beats both continued search and plain rollback in matched-state fork. Repair to GEPA and TextBO-GEPA also helped.
Critical strengths
-
The matched-state fork design is sound. The various techniques like repair, continue, rollback, and stopping are all running from the same saved state, with set minibatch order and randomness. Repair wins 48/48 HPQA forks, 48/48 extraction forks, and decidedly better than rollback by 4.5-6.4 pts on HPQA. I believe this demonsrates that the repair mechanism works. IT's a nice way to eval promp-optimizer interventions than simply comparing end-of-run avgs.
-
Careful eval in general. Data groups are disjoint, test set is used once per run, timing-matched control from separate pilots, and stats robust. Authors are also upfront on where their results are weak, making it much easier for me to judge the paper.
Critical weaknesses
- The paper's data does not support the core claims about RB-Prompt's main components:
- growth trigger: comparing joint trigger against a timing-matched score-only trigger shows 95 CI containing zero in all 9 settings, with only 1 passing the sign-flip test. Untimed score-only repair captures most of the gains anyways.
- bayesian selector (TPE): TPE outperforms uniform random parent selection by at most 0.7 pts. Furthermore, TEXTBO-GEPA plus repair outperforms RB-Prompt in 3 of 9 settings. Overall the growth trigger nor TPE carries the reported gains.
- The reported performance gains aren't fully substantiated, and several simpler alternative explanations are untested. In Appendix A.1, with \beta=0.02, the length penalty can only change a decision once candidates differ in excess length by at least 6.25x the fallback length. So on public benchmark tasks, repair differs from rollback primarily due to a single "Remove or combine rules" rewrite step. The study didn't say whether this rewrite must occur in the search process itself (e.g., by comparing it against a one-shot final prompt consolidation, scheduled consolidation or a length capped baseline like ESPO). Also when repair is disabled while keeping coverage-based selection, the pipeline already outperforms GEPA by 1.3-2.4 pts on HPQA/extraction tasks. So there's baseline discrepancy unexplained, which indicates part of the claimed margin over GEPA is through factors other than repair or TPE. Lastly, despite focusing on instruction growth, the paper didn't report the final prompt length for any evaled method nor provide examples of prompt before/after repair process.
Overall, I think the repair effect is plausible but the paper credit it to parts the data don't yet support, and likely simpler explanations aren't ruled out. TextReg/ESPO already document growth without gains, and ACE's grow-and-refine (https://arxiv.org/html/2510.04618v1) is close to the consolidation rewrite; so what's left is quite incremental.
Questions for authors
-
With repair off and coverage selection, your pipeline beats GEPA by 1.3-2.4 pts on HPQA and extraction. Can you identify where this comes from? Do GEPA/TextBO-GEPA runs score rejected children on the full panel, and get the same demonstration?
-
How many acceptance decisions were modified by the \beta penalty on HPQA and GSM8K? Considering A.1, how should we interpret the -0.68 change when \beta=0 in Table 12?
-
What are the final instruction/prompt length counts across each method listed in Table 2?
Expts that could change my score: 4. Apply A.1 consolidation once to the final top prompts of GEPA (and of RB-Prompt w/o repair), then run normal selection. If this is most of the gains, what does in-search repair add? 5. Periodic consolidation baseline with no detector, and length-capped GEPA.