acceptodds
Withdrawn from ICLR 2027

RB-Prompt: Detecting and Repairing Prompt Saturation in Reflective Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompt optimization adapts large language models to new tasks without changing their weights. Reflective optimizers rewrite prompts using feedback on model errors, but can keep adding instructions after performance stops improving. We call this prompt saturation: search spends evaluation budget on longer prompts with little gain. We address this problem with RB-PROMPT, which detects saturation from recent scores and instruction growth. It returns to a branch’s best saved prompt and asks reflection to remove or combine unnecessary rules. Search continues with a penalty for added instruction length and Bayesian selection guided by past rewrite gains. We evaluate on HotpotQA, GSM8K, and an existing document extraction task at a major global automotive manufacturer, using three model families. Under matched task-evaluation budgets, RB-PROMPT achieves the highest or joint-highest score in all nine settings, with the largest gains on HotpotQA and extraction. It achieves higher scores while using 40–45% fewer optimization tokens than TextBO-GEPA. Comparisons from the same saved states isolate the repair’s benefit over continued reflection. We show that adding repair to GEPA and TextBO-GEPA improves performance without changing their parent-selection rules. These findings support adaptive branch repair as a reusable component of reflective prompt optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automatic prompt optimization, large language models, reflective optimization, prompt saturation, black-box optimization, sample efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_en6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper studies reflective prompt optimizers like GEPA, which suffers from prompt saturation - where the system keeps adding instruction texts to the prompt even when the score has stopped improving. RB-Prompt tackles this by watching each search branch, and flags when a branch when over the last k accepted rewrites the score gain is below a threshold but the instruction grew above another threshold. When a branch is flagged, RB-PRompt that rolls back to the branch's best prompt and ask a reflection model to remove/combine rules, and then applies a relative length penalty to later candidates. Parents are picked with a TPE density ratio over past rewrite gains. Expts use HotPotQA, GSM8K and another proprietary PDF extraction task, and used Claude Sonnet 4.6, GPT-4o and Qwen-2.5-VL-72b, with 16 runs per settings. Their method has the best or tied mean in all nine settings (Tab. 2). It gains 7.4-10.0 pts over GEPA on HPQA, 6.0-10.0 pts on extraction, but only 0.6-1.2 pts on GSM8K. They also show that repair beats both continued search and plain rollback in matched-state fork. Repair to GEPA and TextBO-GEPA also helped.

Critical strengths
  1. The matched-state fork design is sound. The various techniques like repair, continue, rollback, and stopping are all running from the same saved state, with set minibatch order and randomness. Repair wins 48/48 HPQA forks, 48/48 extraction forks, and decidedly better than rollback by 4.5-6.4 pts on HPQA. I believe this demonsrates that the repair mechanism works. IT's a nice way to eval promp-optimizer interventions than simply comparing end-of-run avgs.

  2. Careful eval in general. Data groups are disjoint, test set is used once per run, timing-matched control from separate pilots, and stats robust. Authors are also upfront on where their results are weak, making it much easier for me to judge the paper.

Critical weaknesses
  1. The paper's data does not support the core claims about RB-Prompt's main components:
  • growth trigger: comparing joint trigger against a timing-matched score-only trigger shows 95 CI containing zero in all 9 settings, with only 1 passing the sign-flip test. Untimed score-only repair captures most of the gains anyways.
  • bayesian selector (TPE): TPE outperforms uniform random parent selection by at most 0.7 pts. Furthermore, TEXTBO-GEPA plus repair outperforms RB-Prompt in 3 of 9 settings. Overall the growth trigger nor TPE carries the reported gains.
  1. The reported performance gains aren't fully substantiated, and several simpler alternative explanations are untested. In Appendix A.1, with \beta=0.02, the length penalty can only change a decision once candidates differ in excess length by at least 6.25x the fallback length. So on public benchmark tasks, repair differs from rollback primarily due to a single "Remove or combine rules" rewrite step. The study didn't say whether this rewrite must occur in the search process itself (e.g., by comparing it against a one-shot final prompt consolidation, scheduled consolidation or a length capped baseline like ESPO). Also when repair is disabled while keeping coverage-based selection, the pipeline already outperforms GEPA by 1.3-2.4 pts on HPQA/extraction tasks. So there's baseline discrepancy unexplained, which indicates part of the claimed margin over GEPA is through factors other than repair or TPE. Lastly, despite focusing on instruction growth, the paper didn't report the final prompt length for any evaled method nor provide examples of prompt before/after repair process.

Overall, I think the repair effect is plausible but the paper credit it to parts the data don't yet support, and likely simpler explanations aren't ruled out. TextReg/ESPO already document growth without gains, and ACE's grow-and-refine (https://arxiv.org/html/2510.04618v1) is close to the consolidation rewrite; so what's left is quite incremental.

Questions for authors
  1. With repair off and coverage selection, your pipeline beats GEPA by 1.3-2.4 pts on HPQA and extraction. Can you identify where this comes from? Do GEPA/TextBO-GEPA runs score rejected children on the full panel, and get the same demonstration?

  2. How many acceptance decisions were modified by the \beta penalty on HPQA and GSM8K? Considering A.1, how should we interpret the -0.68 change when \beta=0 in Table 12?

  3. What are the final instruction/prompt length counts across each method listed in Table 2?

Expts that could change my score: 4. Apply A.1 consolidation once to the final top prompts of GEPA (and of RB-Prompt w/o repair), then run normal selection. If this is most of the gains, what does in-search repair add? 5. Periodic consolidation baseline with no detector, and length-capped GEPA.

Reviewer_tvNSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prompt saturation in reflective prompt optimization: accepted rewrites increase instruction length while producing little improvement on a fixed search score panel. RB-Prompt detects this pattern, returns to the best saved prompt on the affected branch, requests a consolidating rewrite, and applies a length-growth penalty during subsequent screening. It also uses a gain-guided TPE parent selector. Experiments cover HotpotQA, GSM8K, and proprietary document extraction with three model families. The strongest evidence comes from matched-state intervention experiments and from adding repair to existing optimizers.

Critical strengths
  1. The paper addresses a practical problem in prompt optimisation: repeated reflection can make prompts longer without making them better. Detecting stalled progress and redirecting the search through repair is a sensible approach.
  2. The repair mechanism can be used with existing optimisers. Its benefits across different models and tasks give the approach practical value beyond the proposed RB-Prompt system.
Critical weaknesses
  1. I'm not yet convinced the full framework is needed. Repair appears useful, but the detector's advantage over timing-matched repair is uncertain, and Bayesian selection adds little over simpler rules.
  2. The definition of saturation seems rather narrow. A search can waste its budget repeatedly proposing candidates that get rejected, yet the detector only tracks accepted rewrites. This leaves a fairly natural failure case uncovered.
  3. Beyond HotpotQA, the evidence is more limited. GSM8K gains are modest, while the extraction results come from a small private dataset. I'd be cautious about drawing broad conclusions from this coverage.
  4. I'm also unsure whether the GEPA comparison preserves its usual screening procedure. If rejected candidates are fully evaluated, fewer rewrites fit into the budget. Until this is clarified, the size of the reported advantage is difficult to assess.
Questions for authors
  1. I think the missing comparison is simply asking for simplification at every rewrite. Did you try that? DLPO already uses simplification, and “Always penalize” in Table 7 seems to be testing something different.

  2. Did GEPA keep its usual minibatch rejection step, or were rejected candidates fully evaluated anyway? I couldn't tell from Appendix A.3.

  3. The β=0.02\beta=0.02 result in Table 12 puzzled me a bit. On the public tasks, batches contain only eight examples and ties already favour shorter prompts. How often does the penalty actually change which prompt gets accepted?

I'd appreciate some clarification, especially on the GEPA comparison, since that affects how I read the results.

Reviewer_VuX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies prompt saturation in reflective prompt optimization: instructions keep growing while progress on a fixed scoring panel becomes small. RB-Prompt detects this pattern using a joint threshold on score improvement over the last k accepted rewrites and instruction-token growth. Once triggered, it returns to the best saved prompt on the affected branch, performs a consolidation-oriented reflective rewrite, and applies an instruction-growth penalty to subsequent screening. A TPE-based selector prioritizes eligible parents using historical rewrite gains. The method is evaluated on HotpotQA, GSM8K, and a proprietary document-extraction workflow using Claude Sonnet 4.6, GPT-4o, and Qwen2.5-VL-72B. Across 16 runs per task-model setting, it obtains the highest or joint-highest mean among the main baselines under matched task-evaluation budgets. The matched-state fork experiments and transplanting repair into GEPA/TextBO-GEPA provide additional evidence for the repair mechanism.

Critical strengths
  1. The study evaluates three model families, multiple prompt optimizers, two public benchmarks, and a real-world document-extraction workflow. The authors carefully account for task-evaluation budgets, separate search and testing data, and report optimization-token consumption. Additionally, incorporating the repair mechanism into GEPA and TextBO-GEPA demonstrates that its benefits are not necessarily tied to the proposed parent-selection strategy.

  2. A major strength is the matched-state experimental design, which compares different interventions from identical saved optimization states under the same subsequent evaluation allowance. Repair consistently outperforms continued rewriting and ordinary rollback on HotpotQA and document extraction. The substantial improvements and positive confidence intervals for these tasks provide meaningful evidence that repair contributes beyond simply restoring an earlier prompt.

Critical weaknesses
  1. The benchmark choice does not test the claimed phenomenon. GSM8K is close to a null control. Accuracy starts at 90–95%, and saturation triggers are rare (Table 11). HotpotQA is run as a single-call task rather than the multi-module programs GEPA targets. The settings where prompt growth is best documented are missing: multi-module tasks such as PUPA and HoVer, and out-of-distribution evaluation.

  2. The gains do not come from the components the method is framed around. Growth-based detection is not significantly better than timing-matched score-only detection in the reported intervals (Table 9). On public tasks, the length penalty almost never changes an acceptance decision (Appendix A.1). The parent selectors differ by at most 0.7% (Table 8). What remains is rollback plus a single “remove/combine rules” rewrite, which yields +6.2% over the fallback on HotpotQA within eight steps, compared with +0.8% for rollback alone (Table 6). The baselines needed to isolate this effect are missing: consolidation without detection, post-hoc consolidation of GEPA’s final prompt, and consolidation without rollback.

Questions for authors
  1. How do the authors distinguish genuine prompt saturation from performance ceilings or answer-formatting effects? Could the method be evaluated on more challenging settings, such as multi-module tasks (HoVer/PUPA) or out-of-distribution benchmarks, where instruction accumulation may be more consequential?

  2. Given the limited gains from growth-aware detection, length regularization, and TPE selection, how much of the improvement is attributable to the consolidation rewrite itself? Could the authors provide matched-budget comparisons with consolidation without detection or rollback and post-hoc consolidation of GEPA’s final prompt?

  3. Since the matched-state experiments demonstrate strong repair gains, but the individual components show limited benefits, what evidence supports the necessity of the full RB-Prompt framework over a simpler consolidation-based approach?

Reviewer_KhEqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RB-Prompt to address prompt saturation in reflective prompt optimization. It detects saturation based on performance stagnation and instruction growth, then repairs affected branches through rollback, LLM-based rewriting, and length regularization. The method also incorporates TPE-based Bayesian parent selection. Experiments across multiple tasks and LLMs demonstrate improvements over existing prompt optimization baselines, with additional analyses evaluating the proposed components.

Critical strengths
  1. The paper addresses prompt saturation in reflective prompt optimization, a practical problem where repeated rewriting increases instruction complexity without meaningful performance improvements.

  2. The paper evaluates the proposed method across multiple tasks and LLMs, with comparisons against several baselines and additional ablation studies. The results demonstrate notable performance improvements, particularly on HotpotQA and document extraction.

Critical weaknesses
  1. The method primarily combines existing techniques, such as heuristic saturation detection, rollback, LLM-based rewriting, and Bayesian selection, making it more of an incremental engineering improvement than a substantial methodological contribution.

  2. The proposed detector relies on two heuristic conditions: limited performance improvement and instruction growth. However, the paper provides insufficient justification for why these two conditions, particularly their combination, can reliably indicate prompt saturation. Similar patterns may arise from different causes, such as instruction conflicts, overfitting, repetitive reflection, or limited search diversity, which are not adequately captured by these simple criteria.

  3. The proposed branch repair returns to the best previously saved prompt and resumes LLM-based rewriting with length regularization. Although the experiments demonstrate performance improvements, the paper does not analyze whether the repaired branches explore substantially different prompts or repeatedly encounter similar stagnation. It therefore remains unclear how the proposed repair changes the original search process beyond restarting it with modified rewriting instructions.

  4. The proposed TPE-based selection relies on four statistical features without sufficiently explaining how they relate to future rewrite gains. Moreover, the experimental improvements over simpler selection strategies are marginal, raising questions about the necessity and effectiveness of the proposed TPE-based parent selection mechanism.

  5. The matched-state experiments (Table 6) only evaluate repair at detected saturation points, without examining whether repair is similarly effective at non-saturated states. Therefore, it remains unclear whether the proposed detector identifies states that particularly benefit from repair. Moreover, Table 9 shows no clear overall improvement over the timing-matched score-only detector.

  6. In Table 6, the comparison between Repair and Rollback simultaneously changes the LLM rewriting instruction and length penalty, making it difficult to identify their individual contributions. Although Table 12 examines the penalty coefficient, the paper lacks a controlled ablation that independently evaluates these two components under the same starting conditions.

Questions for authors
  1. What are the underlying causes of prompt saturation? It may arise from instruction conflicts, overfitting, repetitive reflection, or limited search diversity. These failure modes may require different repair strategies. However, RB-Prompt does not distinguish among them and instead adopts a uniform repair mechanism. Can the authors justify the effectiveness of this approach across different causes of saturation?

  2. How does Branch Repair help the optimizer escape previously stalled search regions rather than repeatedly exploring similar prompts?

  3. Could the authors provide a stronger justification for the heuristic design choices, particularly the use of a uniform repair strategy for different causes of prompt saturation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.