acceptodds
Under review as a conference paper at ICLR 2027

APEX-Data: Automated Prompt Engineering eXpert with Dynamic Data Selection

Abstract

Large Language Models are highly sensitive to prompt formulation, necessitating automatic prompt optimization to unlock their full potential. While evolutionary algorithms have emerged as the dominant paradigm, they suffer from a critical bottleneck: data efficiency. Current methods treat the development dataset as a static benchmark, wasting significant compute budget on uninformative data. In this work, we introduce APEX-Data (Automatic Prompt Engineering eXpert with Dynamic Data Selection), a novel framework that optimizes the data usage alongside the prompt search. APEX dynamically stratifies the dataset into Easy, Hard, and Mixed tiers based on the optimization lineage. By prioritizing the Mixed tier, which identifies the data where the LLM has mixed performance, we identify two high-leverage subsets: the addressable frontier for generating informative mutations and the rank-sensitive frontier for distinguishing candidate quality. We evaluate APEX across three diverse benchmarks: IFBench, SimpleQA Verified, and FACTS Grounding. Under a fixed budget of 5,000 evaluation calls, due to its data efficiency, APEX outperforms the initial prompt by an average of 11.2% on Gemini 2.5 Flash and 6.8% on Gemma 3 27B, demonstrating that a data-centric approach is key to efficient and effective prompt optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
promp optimization, data selection, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C2oNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces APEX-DATA (Automated Prompt Engineering expert with Dynamic Data Selection), a data-centric framework designed to improve the data efficiency of evolutionary prompt optimization under black-box API constraints. Rather than treating the development set as a static benchmark evaluated uniformly, APEX-DATA dynamically stratifies examples into three tiers—Easy, Hard, and Mixed—based on pass/fail outcomes recorded across a sliding lookback window (k=5) of recent prompt variants in the optimization lineage.

The framework concentrates evaluation and generation compute on the volatile Mixed tier through two primary mechanisms:

  1. Trajectory-Guided Mutation: Conditions prompt edits on recent failures drawn from the Mixed tier (B_{M,0}) rather than intractable Hard errors (B_{H,0}), targeting fixable regressions while maintaining a usage history cache to prevent repetitive cycling.
  2. Rank-Sensitive Selection: Evaluates candidate prompts against the incumbent prompt using a budget-capped evaluation set D_eval. It prioritizes volatile unknown examples (B_{M,empty}) and employs an annealed anchor-ratio schedule (alpha_t) to balance positive stability checks (B_{M,1}, B_{E,empty}) against negative error corrections (B_{M,0}, B_{H,empty}).

Experiments across three diverse tasks and evaluation regimes—IFBench (programmatic verifier), SimpleQA Verified (exact match), and FACTS Grounding (LLM-as-a-judge)—demonstrate that APEX-DATA outperforms initial prompts by +11.2% on Gemini 2.5 Flash and +6.8% on Gemma 3 27B under a fixed global budget of 5,000 evaluation calls, surpassing baselines such as APO, MIPROv2, TextGrad, and GEPA. Ablation studies validate the utility of finite lookback sizing, Mixed-tier prioritization, and pairwise ranking agreement.

Critical strengths
  • Conceptual shift to data-centric optimization: The paper shifts the focus of automated prompt engineering from designing complex meta-prompts and search heuristics to optimizing the informativeness of the evaluation data. Reframing the development set from a static benchmark into a dynamic pool of informative signals directly tackles the primary compute bottleneck in black-box prompt search.
  • Principled operationalization of frontiers: Categorizing volatile instances into an "addressable frontier" for mutation (targeting solvable regressions) and a "rank-sensitive frontier" for selection (isolating candidate-discriminating instances) is mathematically straightforward, clean, and intuitive. The adaptive anchor schedule sensibly balances positive preservation against error correction as prompts mature.
  • Broad empirical coverage across evaluation regimes: The framework is evaluated across three distinct task archetypes and verification protocols: verifiable programmatic constraints (IFBench), gold-answer factuality (SimpleQA Verified), and model-based evaluation (FACTS Grounding), showing consistent improvements across both proprietary (Gemini 2.5 Flash) and open-weight (Gemma 3 27B) models.
  • Informative ablation and analytical experiments: The analyses in Section 4.3 thoroughly validate the core assumptions: Figure 4 verifies that mutating on Hard errors actively degrades performance (-8.2%), Figure 5 confirms substantial improvements in pairwise ranking agreement over uniform random sampling, and Table 4 shows that performance gains are not confounded by prompt length expansion.
Critical weaknesses
  • Budget accounting fairness and caching parity: APEX-DATA relies heavily on incremental caching (Algorithms 1 & 2), evaluating only candidate prompts on historically evaluated instances while reusing incumbent results. It is not made clear whether baselines (such as APO, GEPA, and MIPROv2) were given identical caching infrastructure and whether the 5,000-call budget was calculated identically across all methods. If baselines were re-evaluating incumbents or lacking cache reuse, APEX-DATA would effectively enjoy far more search iterations within the same 5,000 evaluation calls. In addition, it is unclear whether the initial full evaluation of P_0 (Algorithm 2, line 2, which requires 500-700 calls) is subtracted from the 5,000 budget.
  • Under-specification of cold-start and early-iteration tiering: Tier assignment depends on the last k valid outcomes (R_i). At iteration t=1, every evaluated example has exactly one evaluation (|R_i|=1), meaning its outcome set is either {1} or {0}. Consequently, the Mixed tier is strictly empty (B_{M,s} = empty), D_req is empty, and the pass rate ratio rho_mix in Eq. (10) results in division by zero. The manuscript does not specify how the algorithm bootstraps when t < k or how early cold starts are handled.
  • Susceptibility to LLM-judge stochasticity: On FACTS Grounding, binary scores are generated by an LLM-as-a-judge. Because LLM judges exhibit inherent stochasticity and non-determinism, a single random output variation can spuriously push an example from Easy or Hard into the Mixed tier. The paper does not analyze the stability of lineage-based tiering under evaluator noise, nor does it specify whether temperature was set to 0 or multiple judge samples were averaged.
  • Inconsistency regarding Hard error sampling: While Section 4.3 and Figure 4 clearly demonstrate that prioritizing Hard instances degrades performance by -8.2% relative to the initial prompt, Eq. (8) and Algorithm 2 line 5 still explicitly sample mutation errors from (B_{M,0} U B_{H,0}) \ U. The paper lacks an ablation analyzing why B_{H,0} is included and what occurs when mutations are restricted strictly to B_{M,0}.
  • Lack of statistical significance and run reporting: Table 1 presents results as mean +/- std, but never specifies the number of independent random seeds or runs conducted. Given that several improvements on Gemma 3 27B are moderate (e.g., 93.3 +/- 0.1 vs. GEPA's 91.7 +/- 0.5 on FACTS Grounding), explicit reporting of sample sizes and statistical significance tests (e.g., p-values) is needed to confirm robustness.
Questions for authors
  1. Budget and Caching Parity: Did all baseline implementations (APO, GEPA, MIPROv2, TextGrad) utilize the exact same incremental caching mechanism as APEX-DATA? Specifically, does the 5,000-call global budget account for the initial full evaluation of P_0 on dataset D, and does an evaluation call refer strictly to task-model inferences or also LLM-judge calls?
  2. Cold-Start Initialization: How does the framework handle early iterations (t < k), especially iteration t=1 where |R_i|=1 and the Mixed tier is empty? How are D_req, rho_mix, and candidate selection computed when no Mixed instances exist?
  3. Ablation on Hard Error Inclusion: Given that Figure 4 indicates Hard error prioritization severely degrades performance (-8.2%), why does Eq. (8) sample from both B_{M,0} and B_{H,0}? What proportion of sampled mutation batches ended up containing Hard errors, and have you evaluated performance when restricting mutations strictly to B_{M,0}?
  4. LLM-Judge Stochasticity: On FACTS Grounding, how did you control for evaluator variance (e.g., judge temperature, majority voting)? Did judge noise cause false positives in the Mixed tier, and how resilient is the lineage window to such classification errors?
  5. Experimental Seeds and Significance: How many independent runs were performed to compute the mean and standard deviations in Table 1? Are the performance gains over GEPA on Gemma 3 27B statistically significant?
  6. Meta-Optimizer Overhead: What is the total token consumption and API cost incurred by the meta-optimizer LLM (Critique and Mutate prompts in Listings 4 & 5) and history bookkeeping across the full 5,000 evaluation budget compared to baselines?
2 comments

We thank the reviewer for the detailed and constructive review. In response, we show that the budget and caching accounting is fair, specify the cold-start fallback, correct Eq. 8, measure the judge's reliability, and report run counts and significance tests.


W1 / Q1: Budget accounting and caching parity

It is not made clear whether baselines … were given identical caching infrastructure and whether the 5,000-call budget was calculated identically across all methods. ... it is unclear whether the initial full evaluation of P0 … is subtracted from the 5,000 budget.

The comparison is fair: every method is charged for the same unit under the same 5,000-call budget, and every baseline runs with its official implementation, including its own caching.

Call budget: one task-model inference plus its scoring (including judge calls), defined the same way for all methods.

P0 evaluation: the initial full evaluation of P0 is counted within the 5,000 budget. Cached retrievals are not counted, for any method.

Baselines and caching:

  • APO uses its official cached scorer: a (prompt, instance) score is computed once and reused. Its UCB evaluation samples random minibatches, so repeated pairs are uncommon.
  • GEPA stores the validation scores of every candidate and reuses them when comparing candidates on its Pareto front.
  • MIPROv2 and TextGrad evaluate candidates on random minibatches.

Incremental evaluation is a design choice and part of APEX-DATA's contribution. It pays off because APEX-DATA's data selection repeatedly targets the same informative instances, so the current best prompt's results can be reused and only the new candidate needs inference. With random sampling, as in the baselines, there is little to reuse.


W2 / Q2: Cold start

At iteration t=1, … the Mixed tier is strictly empty …, and the pass rate ratio rho_mix in Eq. (10) results in division by zero. How are D_req, rho_mix, and candidate selection computed when no Mixed instances exist?

Thank you for catching this missing detail. We will add the rule to Alg. 2 and §3.1.

  • Fallback at the start. Tiers use the last min(k, |Hi|) valid outcomes. At t = 1 the Mixed tier is empty, so mutation errors are drawn from Hard failures, and the selection budget is filled from the other tiers (D_req = ∅, R = N). When no Mixed instance has a known outcome, ρ_mix falls back to ρ_all, the current prompt's overall pass rate, so the positive share in Eq. 10 is min(α, ρ_all).
  • Cold start is short. As Fig. 6 shows, the Mixed tier grows quickly within the first few iterations, so cold start affects only the initial iterations.

W3 / Q4: LLM-judge stochasticity on FACTS Grounding

Susceptibility to LLM-judge stochasticity: On FACTS Grounding, binary scores are generated by an LLM-as-a-judge … how did you control for evaluator variance (e.g., judge temperature, majority voting)? Did judge noise cause false positives in the Mixed tier

Judge protocol: We follow the official FACTS Grounding judge templates and two-stage pipeline (Jacovi et al., 2025), whose templates were selected against human-annotated held-out examples. Each item receives multiple judge calls, including a quality filter, and grounding check sentence by sentence. The item passes only if all checks pass. The judge (Gemini 2.5 Flash, temperature 0) is the same for every method.

Measured reliability: We re-judged fixed responses on all 200 test items three times each. Verdicts were identical across all three repeats for 98% under the optimized prompt. Judge noise is therefore small.


W4 / Q3: Eq. 8 Clarification

Eq. (8) … sample mutation errors from (B_{M,0} U B_{H,0}) \ U. What proportion of sampled mutation batches ended up containing Hard errors, and have you evaluated performance when restricting mutations strictly to B_{M,0}?

We thank the reviewer for catching this inconsistency. As stated in the paper text (line 216, Fig. 4), our method samples mutation errors in priority order: unvisited Mixed failures first, and Hard failures only as a fallback when no unvisited Mixed failures remain. Eq. 8 and Alg. 2 did not reflect this priority, and we will correct both.

Hard errors are rare in mutation batches: across runs, ~10% of mutation batches contain any Hard item, mostly at cold start, when the Mixed tier is empty by construction.

The effect of each tier is already in Fig. 4. Prioritizing Mixed gives +13.8, All +4.4, Easy −1.1 and Hard −8.2. The −8.2% corresponds to prioritizing Hard instances, a different regime from using them as a rare fallback.

W5 / Q5: Number of seeds and statistical significance

Table 1 presents results as mean +/- std, but never specifies the number of independent random seeds or runs conducted. ... explicit reporting of sample sizes and statistical significance tests (e.g., p-values) is needed to confirm robustness.

Are the performance gains over GEPA on Gemma 3 27B (e.g., 93.3 +/- 0.1 vs. GEPA's 91.7 +/- 0.5 on FACTS Grounding) statistically significant?

Table 1 reports mean ± std over 3 independent runs per cell. We will state this in the caption and refine the claim to be more precise: "APEX-DATA performs better in most settings and is comparable to the strongest baseline in the others." No single baseline is consistently best. The strongest baseline changes across cells (APO, TextGrad, GEPA).

Welch's t-test of APEX-DATA against every baseline in every cell of Table 1: APEX-DATA's mean is at or above every baseline in all 24 comparisons, and significantly better in 18 of 24 comparisons (one-sided, p < 0.05; 16 of 24 two-sided), and never worse.. On Gemma FACTS, the gain over GEPA is significant (p = 0.014).


Q6: Meta-optimizer overhead

What is the total token consumption and API cost incurred by the meta-optimizer LLM?

APEX-DATA's optimizer overhead is small, 0.1M to 0.5M tokens across models and benchmarks, at most 6% of its total token cost. For comparison, baselines use up to 2.8M (APO), 1.9M (MIPROv2), and 0.9M (TextGrad), with only GEPA lower (up to 0.3M).

Reviewer_JCTDOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In my understanding, the main contribution of this paper is in prompt engineering (automatic prompt optimization). The paper finds that Mixed examples are more useful. Mixed examples are examples that recent prompts sometimes answer correctly and sometimes answer incorrectly. Examples that all recent prompts answer correctly have no error to fix. The old prompt and the new prompt also give the same result on these examples. Thus, more evaluation on them only wastes budget. Examples that all recent prompts answer incorrectly are not only useless for prompt revision. They are also harmful, because they move the optimization in a wrong direction. Mixed is not a fixed property of an example. It changes as the prompt changes during the optimization.

Critical strengths
  1. The figures are clear. The method is simple and intuitive. It is also practical. The method needs only the pass/fail result of each example. It does not need logits, human labels, or an additional model. The method itself only records pass/fail results and adds a cache. You can add it directly to existing frameworks such as APO and GEPA.

  2. The method gives improvements on different types of tasks (instruction following, factual question answering, and grounded generation).

Critical weaknesses

My overall view is that the main idea is not new. The idea is the same as uncertainty sampling in active learning and as curriculum learning: select the most informative examples. Diao et al. (2024) and Dong et al. (2025) also use similar methods. Their methods to measure information are different from the method in this paper. But the paper should at least compare with them as baselines. Thus, I have doubts about the baselines and about the relation to this similar prior work.

[1] Diao et al., Active Prompting with Chain-of-Thought for Large Language Models, ACL 2024.

[2] Dong et al., Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization, Findings of ACL 2025.

Questions for authors

Q1. Is the method still effective with stronger and newer models? My overall feeling is that prompt engineering becomes less important when models have stronger context understanding ability. The experiments use only Gemini 2.5 Flash and Gemma 3 27B. With newer models, for example GPT-6 or Claude Opus 5.5, the result can be different.

Q2. What happens if the share of Mixed data is too small? In that case, there is too little signal to update.

Q3. Figure 6 shows no clear trend in the share of Mixed data. Also, as the optimization iterations increase, I expect the share of Easy data to increase since prompt would be better. Why does Figure 6 not show this?

3 comments

We thank the reviewer for the positive assessment and the helpful questions.


W1: Novelty

My overall view is that the main idea is not new. The idea is the same as uncertainty sampling in active learning and as curriculum learning: select the most informative examples. Diao et al. (2024) and Dong et al. (2025) also use similar methods. ...

We agree that APEX-DATA shares the high-level principle that data properties drive progress. Our contribution is answering three questions that were open for prompt optimization:

  1. How much does data matter in prompt optimization? Will it really drive the performance?
  2. What are informative data for prompt optimization? What signals to use?
  3. How to efficiently select data in practice? How to design a method?

Existing methods from these areas do not transfer directly. Prompt optimization changes the problem itself: the learner is a sequence of discrete prompts on a black-box LLM, which offers no confidence scores to measure uncertainty, and instance difficulty shifts with every mutation, so no fixed curriculum applies. Answering the three questions above therefore requires new signals rather than a transfer of existing ones.

APEX-DATA (ours)Active learning (uncertainty sampling)Curriculum learning
Informative dataHigh variance in performance across the prompt lineageLow prediction confidence / high model entropyEasy examples first, harder later
DetectionLookback over the black-box evaluation history of recent promptsInternal model confidence or entropyA predefined difficulty measure or schedule
UtilizationTrajectory-guided mutation and rank-sensitive selectionTriggering human annotation on selected dataOrdering the training data
LearnerA sequence of discrete prompts (black-box LLM)A parametric modelA parametric model

Both works (Diao et al. 2024 and Dong et al. 2025) are discussed in our related work (§5). They rest on different problem definitions from APEX-DATA and are not comparable:

  • APEX-DATA is designed for automatic prompt optimization for black-box LLMs on broad generation tasks.

  • Diao et al. (2024), Active-Prompt, is not an automatic prompt optimization method, but selects which questions to annotate as few-shot chain-of-thought exemplars. Its problem setup differs from APEX-DATA as the goal is to select data to annotate by humans with no optimization loop.

  • Dong et al. (2025), IPOMP, is designed for prompt optimization for classification tasks with a fixed label set, and requires sentence embeddings and output-label logits.

Neither can be run under our black-box, generative-task protocol without changing its core assumptions. The closest baselines are our Stratified and Diversity Prior selectors (Fig. 3), at 43.5 and 44.9 vs. 52.3 for APEX-DATA. We will expand the related-work paragraph with these specifics.


Q1: Stronger LLM

Is the method still effective with stronger and newer models? ...

We ran APEX-DATA and the two strongest baselines from Table 1, APO (|D|, best on Flash IFBench) and GEPA (best on average), with Gemini 4 Hydrogen (released November 2026, among the top models on the Artificial Analysis Intelligence Index) as both task and optimizer model on IFBench (same protocol and 5,000-call budget).

MethodTest acc
Initial prompt74.8
APO (|D|)79.3
GEPA79.6
APEX-DATA80.6

Prompt optimization still helps the stronger model, and APEX-DATA remains ahead of both baselines. The stronger model starts much higher, so the absolute gain is smaller, but APEX-DATA closes a similar share of the remaining headroom as on Gemini 2.5 Flash. Prompt optimization therefore remains important even for frontier models.


Q2: Mixed data size

What happens if the share of Mixed data is too small? In that case, there is too little signal to update.

We address this concern from three perspectives:

  1. Fallback by design. If few Mixed failures are available, mutation falls back to Hard failures and the selection budget is filled from the other tiers (Alg. 2, lines 12 and 13), so optimization never stalls.
  2. Mixed data arises naturally while there is headroom. As the prompt improves, each new candidate fixes some instances and breaks others, which creates new mixed data along the lineage.
  3. In practice the Mixed share is large enough. It rises quickly in the first iterations and stabilizes around 20 to 25% of the dataset (Fig. 6).

Q3: Trend of Mixed and Easy share

Figure 6 shows no clear trend in the share of Mixed data. Also, as the optimization iterations increase, I expect the share of Easy data to increase since prompt would be better.

The tier shares do change over the run, just gradually rather than sharply.

Mixed share: Instances keep moving between tiers. Instances move from Hard to Mixed as prompts start solving them, and from Mixed to Easy once they are reliably solved. Inflow and outflow roughly balance, so the Mixed share holds around 20 to 25% (Fig. 6) after the initial stage.

Easy share: It grows gradually, rising from about 40% of the training data at the start to about 50% at the end, with roughly 5 points of that gain after iteration 10. Growth is gradual because an instance counts as Easy only if all of the last k = 5 prompts pass it, and each new mutation can move previously passing instances back to Mixed.

Thanks for the response. This address my concerns. I would keep my supportive score.

Reviewer_CEgpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

APEX-DATA is a data-centric prompt optimizer. Its claim is that the binding constraint on evolutionary APO is not the search algorithm but the dev set: current methods treat the dataset as a static ruler, spending budget on examples that are either already solved by every candidate or unsolvable by any of them, so both mutation and selection run on signal-free data. The proposed fix is to re-price each example every iteration according to its history with the current prompt lineage.

Mechanically: the run maintains a sparse score matrix over prompts x examples. For each example, the last k outcomes under prompts that actually evaluated it (k=5 default) define a local history; all-ones makes it Easy, all-zeros Hard, mixed {0,1} Mixed. Crossing that tier with the current prompt's outcome yields nine buckets, of which two matter. Mutation draws its error batch from Mixed-and-currently-failing (plus Hard failures) with a usage log preventing repeats, on the theory that these are "soft failures" the model has recently been able to solve, and routes them through a structured Critique (Locator / Diagnosis / Instruction) then a constrained Mutate editor (variable lockdown, logic lock, edit only near the locator). Selection spends a fixed 100 evaluation calls per iteration: a mandatory set of historically-mixed instances that were skipped last round, then an anchor ratio alpha split between a positive set (catch regressions) and a negative set (confirm fixes), with alpha annealed upward on each accepted candidate and clamped by the mixed-tier and overall pass rates. Evaluations are cached, so the current prompt is never re-run on instances already scored; a candidate replaces the incumbent only if it wins on that paired evaluation set. Budget is a global 5,000 evaluation-function calls.

Contributions

  1. Names a real and comparatively neglected bottleneck — data efficiency in APO — and argues the informative signal lives in a dynamically shifting mixed tier rather than in a fixed split.
  2. A concrete algorithm operationalizing that: lineage-based tiering over a lookback window, an "addressable frontier" for mutation, a "rank-sensitive frontier" for selection with annealed anchors and incremental caching (Algorithms 1-3, full prompt templates in Appendix D).
  3. Two frontier constructs plus the components that use them, presented as a general, model-agnostic, black-box-compatible layer that can sit under existing optimizers.
  4. Empirical claim: at a fixed 5,000-call budget, +11.2 points over the initial prompt on Gemini 2.5 Flash and +6.8 on Gemma 3 27B across three benchmarks with three different evaluation protocols.

Validation Setting: IFBench (500 optimize / 294 test, disjoint constraints), SimpleQA Verified (700/300), FACTS Grounding (660/200, LLM-as-judge); Gemini 2.5 Flash and Gemma 3 27B, same model as both target and meta-optimizer; baselines Initial, APO at |D| and |D|/2, MIPROv2, TextGrad, GEPA, official implementations at default settings, all under a common 5,000 evaluation-call budget.

Main table: Gemini 49.3 -> 60.5 aggregate, best baseline GEPA 54.5 (so +6.0 over the strongest baseline, +11.2 over the seed); Gemma 41.2 -> 48.0, best baseline APO 45.2 (+2.8 over baseline, +6.8 over seed). Wins in every reported cell, but MIPROv2 and TextGrad are absent from the Gemma block entirely, and Gemma/SimpleQA is a tie with APO at 11.5.

Mechanism evidence, all on IFBench + Gemini: a selection-strategy ladder under an otherwise identical framework ; tier-priority sweep ; pairwise ranking agreement against full-set ground truth as a function of subset size, which tests the rank-sensitivity premise directly rather than through end-task score; tier composition over iterations with Mixed stabilizing at 20-25%; a 2x2 ablation of mutation-side and selection-side components ; budget curves; prompt-length comparison ; accuracy-token Pareto plot; lookback sensitivity (50.3 / 52.3 / 50.6) and a qualitative prompt-evolution walkthrough.

the diagnostic is correct and orthogonal to the search-side line of work, and the Fig. 3 ladder plus Fig. 5 rank-agreement experiment are the right way to argue it — they isolate data choice from iteration count and test the premise rather than only the endpoint. The weaknesses are in how far the evidence is pushed: the quoted headline is against the seed prompt rather than the state of the art (+6.0 and +2.8), the Gemma comparison is partial, the budget counts only evaluation-function calls so the two meta-LLM calls per iteration and any judge calls go uncounted while the method's whole advantage is buying more iterations with cheaper decisions, no principled uncertainty-sampling baseline appears despite the closest prior work being dismissed in one clause, all mechanism analysis is single-benchmark single-model without variances or a stated number of seeds, and the three tasks all reward output-contract and brevity behavior rather than reasoning, which is where the tier logic would be most at risk.

Critical strengths

First, the paper isolates its causal claim with the right experiment, and that experiment produces a gap too large to be noise. Fig. 3 holds the framework, the budget, and the iteration count fixed and varies only how evaluation instances are chosen: random 42.9, stratified 43.5, diversity prior 44.9, no-lookback 49.3, full 52.3. The 9.4-point spread between random selection and the full method is larger than the spread between the named baselines in Table 1, and the 3.0-point step from k=+∞ to k=5 tests the specific claim — that the window, not just the tiering, is doing work. Why this is consequential to me: it is the one place where my sharpest objection (that the gains might be an artifact of buying more iterations with cheaper per-decision evaluation) is neutralized, because iteration count is held constant in that comparison. It also means the contribution is not reducible to a knob on an existing optimizer. When I discount the headline, I am left with a controlled result showing that the choice of evaluation data moves final accuracy more than the choice among published search algorithms — that is a claim about the field, not just about this method, and it is worth publishing even if the 11.2-point framing is walked back.

Second, Fig. 5 validates the mechanism at the level of the premise rather than the endpoint. Measuring pairwise rank agreement against full-set ground truth as a function of subset size tests whether a mixed-tier subset of 50–100 examples actually preserves candidate ordering, which is precisely the assumption the selection stage rests on, and it does so without depending on any end-task score. Why this is consequential: it converts a plausible-sounding design intuition into a measurable property, and the measurement is cheap enough to become a standard diagnostic for this literature. It also gives me a falsifiable handle — if a future paper finds that disagreement-based subsets do not preserve ordering on its task, the whole argument fails cleanly. Very few prompt-optimization papers offer a check that can fail in an informative way; most offer only final accuracies. Together with the ablation ladder, this is what makes me willing to treat the paper as a contribution rather than a hyperparameter study, despite the budget accounting and the single-benchmark scope of both analyses.

Critical weaknesses

Weakness 1 — the efficiency claim is denominated in a currency that excludes the cost the method spends.

The 5,000-call budget counts evaluation-function calls only. Outside that ledger sit the two meta-LLM calls per iteration (Critique, then Mutate), the LLM judge on FACTS, and the question of whether cache hits are billed at all. This is not a bookkeeping nit, because APEX's mechanism is precisely to convert a smaller per-decision evaluation into more optimization steps: with N=100 per iteration against a 500-instance dev set, it runs roughly five times as many mutate-and-compare rounds as GEPA does at the same nominal budget, and each round costs two long-context calls (the critique is fed error cases, the mutation is fed the full prompt). The paper's headline framing — data efficiency, "wasting significant compute budget," "maximizes the optimization signal per evaluation call" — is a total-compute claim, but its measurement is a task-model-only claim. Fig. 8's accuracy-versus-token plot is the place this would be settled and the counting rule is never stated, so I cannot tell whether the two methods are being compared on the same quantity. Consequential because it is not a missing number, it is a possible inversion: if total cost including meta-calls is higher for APEX, then the correct description of the contribution is "spend more optimizer compute to buy more iterations, and spend it on better-chosen data," which is a real but much narrower and much more familiar claim than the one in the abstract, and it would require the efficiency framing throughout to be rewritten. Everything else I could ask for here — the Gemma baselines, the seed counts, the variance on the analysis figures — is a gap I can hold open; this one has the potential to change what the paper is about.

Weakness 2 — the evidence base cannot support the generality the paper claims, and the nearest prior work is dismissed rather than compared against.

Three benchmarks, all of which reward output-contract and constraint-compliance behavior: IFBench verifies constraints programmatically, SimpleQA is short-form factuality where brevity helps, FACTS is grounded long-form judged by an LLM. Not one is a reasoning task — no math, no multi-hop, no code — even though that is where prompt optimization is most valuable and where per-instance binary pass rates are least informative, because there the difference between two candidate prompts is rarely "solved versus unsolved" on a given instance but "right path versus plausible wrong path." All mechanism evidence (Figs. 2–6, Tables 2–3) is a single benchmark on a single model. The second model panel is incomplete in a way that matters: MIPROv2 and TextGrad are absent from the Gemma block entirely, and on the cells that remain the margin is +2.8 over the best baseline with an exact tie on SimpleQA (11.5 vs 11.5). So "consistently outperforms baselines across both LLMs" is, in evidence, one model with a clear lead and one model with a partial table and a small one. Meanwhile the closest neighbor — performance-guided selection of evaluation data for prompt optimization — is dispatched in a single subordinate clause asserting reliance on logits and classification tasks, with no citation-level verification and no experimental comparison, and no principled disagreement-based or dueling-comparison sampler appears as a baseline at all. Consequential because these two things are exactly what the paper's contribution list asserts: contribution 1 is the general diagnosis of a field-wide bottleneck, contribution 3 is breadth of validation. If the effect is specific to constraint-following benchmarks and to a Flash-class meta-optimizer, and if the idea overlaps substantially with existing work on evaluation-data selection, then what remains is a well-executed engineering recipe for one class of task rather than a paradigm shift — and the paper's own framing invites the stricter reading.

Questions for authors
  1. Please give the total cost ledger per benchmark and model, not just evaluation-function calls: meta-LLM (Critique + Mutate) calls and tokens, FACTS judge calls, and task-model calls, for APEX and for each baseline, plus whether Fig. 8's token axis includes meta-calls. Related: do cached retrievals count against the 5,000, and do the baselines also benefit from the incremental cache, or only APEX? If, on a total-token or total-dollar basis, APEX is at parity or cheaper, my central objection dissolves and I will say so in my final review. If it is costlier, the efficiency framing has to be rewritten around "cheaper decisions, more iterations," and I need to know that before I decide rather than after.

  2. What separates your Mixed tier from plain disagreement-based (uncertainty) sampling over the current population? Concretely: if I take candidate prompts from the existing population, score instance-level disagreement among them, and sample 100 instances with the highest disagreement — no lookback window, no Easy/Hard/Mixed trichotomy, no alpha annealing — does that land near 52.3 or near 43.5? You already have the machinery to run this, and it is the baseline a reviewer will reach for first. If it ties with APEX, the contribution is a specific and rather elaborate implementation of a known principle; if it trails, the lineage windowing is doing real work and the paper is stronger than I currently think.

  3. Can you report the closest prior work's protocol and why it is not comparable? Specifically Dong et al. 2025 on performance-guided evaluation-data selection for prompt optimization, and Diao et al. 2024 on active prompting. The current sentence asserts reliance on internal signals and classification-only scope; please substantiate that with what those papers actually assume, or run one of them. This determines whether I treat the novelty claim as established or as unverified, and it goes directly to my rejection case.

  4. Can you fill in the Gemma block — MIPROv2 and TextGrad — and tell me what "consistently outperforms baselines across both LLMs" means given that Gemma/SimpleQA is 11.5 vs 11.5 against APO and the aggregate margin over the best baseline there is 2.8 points? If those two baselines were run and are simply omitted from the table, that is an easy fix; if they were not run, the cross-model claim is half-tested.

2 comments

We thank the reviewer for the detailed review and the clear criteria. In short: following the criterion in the review, on a total-token basis including all meta-LLM and judge calls, APEX-DATA is among the most cost efficient methods. The disagreement baseline the reviewer proposed reaches 42.9, well below APEX-DATA (52.3).


W1 / Q1: Total cost including meta-LLM and judge calls

The 5,000-call budget counts evaluation-function calls only. Outside that ledger sit the two meta-LLM calls per iteration …, the LLM judge …, and the question of whether cache hits are billed at all. ... Please give the total cost ledger ... do the baselines also benefit from the incremental cache, or only APEX? If, on a total-token or total-dollar basis, APEX is at parity or cheaper, my central objection dissolves

We thank the reviewer for the clear criterion. We re-ran all methods with per-call logging of every task-model, optimizer, and judge call. On a total-token basis, APEX-DATA is among the most cost efficient methods. Its optimizer overhead is small (0.1M to 0.5M tokens, at most 6% of its total) and does not cause the inversion.

Total tokens during optimization (task + optimizer + judge, excluding test-set evaluation):

MethodFlash IFBenchFlash SimpleQAFlash FACTSGemma IFBenchGemma SimpleQAGemma FACTSAverage
APO (|D|)11.8M15.9M71.4M7.5M10.0M82.5M33.2M
MIPROv213.9M10.4M84.6M14.9M9.1M81.3M35.7M
TextGrad13.4M19.2M58.9M6.4M10.0M53.0M26.8M
GEPA9.6M11.9M70.8M6.9M14.4M76.9M31.8M
APEX-DATA8.1M10.8M64.1M8.8M8.5M60.5M26.8M

We will replace Fig. 8 with this full ledger table in the revision.

Cached retrievals. The 5,000 budget counts actual task-model inferences, including the initial full evaluation of P0. Cached retrievals are not counted as they do not cost tokens. All baselines run in their official configuration with their own caching where it exists (e.g., APO caches scores by prompt and data, GEPA reuses stored validation scores). Incremental evaluation is part of APEX-DATA's contribution: its data selection deliberately re-targets informative instances, so the current best prompt's results can be reused.


W2: More Results

Three benchmarks, all of which reward output-contract and constraint-compliance behavior ... Not one is a reasoning task ... MIPROv2 and TextGrad are absent from the Gemma block …

Missing Gemma results: we have now run them under the same protocol with 3 seeds:

Gemma 3 27BIFBenchSimpleQAFACTS GroundingAvg
Initial prompt33.49.480.741.2
APO (|D|)35.711.588.545.2
MIPROv236.410.089.545.3
TextGrad23.410.691.241.7
GEPA34.19.491.745.1
APEX-DATA39.311.593.348.0

With the complete Gemma results, APEX-DATA is best or tied for best on every Gemma benchmark, and best on average.

We will revise the "consistently outperforms" claim to: "APEX-DATA performs best in most settings and is comparable to the strongest baseline in the others. No single baseline is consistently second."

More benchmarks: We agree that adding math and coding benchmarks would be beneficial, and we appreciate the reviewer's openness to this gap. We will state it as a limitation and future work in the revision. Two clarifications on scope:

  1. Method-wise, APEX-DATA only needs per-instance outcomes from a black-box model's evaluator (a programmatic check, a reference match, or an LLM judge). Reasoning tasks with checkable final answers provide such outcomes, and nothing in the method is specific to constraint following.
  2. Benchmark-wise, the three benchmarks are not purely format checks: IFBench requires jointly satisfying several verifiable, out-of-distribution constraints, SimpleQA tests short-form factuality from parametric knowledge, and FACTS Grounding requires long-form generation to be faithful and of high quality.

Q2: Plain disagreement-based sampling baseline

What separates your Mixed tier from plain disagreement-based (uncertainty) sampling over the current population? ...

Experiment (as the reviewer specified): We maintain a population of 3 candidate prompts, score instance-level disagreement over their recorded outcomes, and select the top-100 instances, with no lookback window, no tiers and no α. Same budget (5,000 calls), IFBench / Gemini 2.5 Flash.

Data selection strategyTest acc
Initial prompt38.5
Stratified43.5
Population disagreement42.9
APEX-DATA52.3

Disagreement sampling baseline is well below APEX-DATA. As the reviewer anticipated, APEX-DATA's dynamic data selection does real work beyond the generic disagreement principle.

Why plain disagreement falls short: it has two structural problems

  • Maintaining a population costs search depth. Population disagreement requires more than one prompt in the pool, and every member must be scored on each selected set. Under the same budget, the disagreement baseline completes only about two thirds of APEX-DATA's iterations.
  • The plain population disagreement set is small. It needs instances that all members have been scored on, and with 100 instances evaluated per iteration, only about 10% of the dataset shows disagreement. The rest is filled at random. APEX-DATA instead considers each instance's most recent (lookback) scores along the lineage (Eq. 6), so its Mixed tier accumulates over the whole search and covers 20 to 25% of the dataset (Fig. 6).

Q3 / W2c: Prior work protocol

Meanwhile the closest neighbor is dispatched in a single subordinate clause asserting reliance on logits and classification tasks … Can you report the closest prior work's protocol and why it is not comparable? Specifically Dong et al. 2025 on performance-guided evaluation-data selection for prompt optimization, and Diao et al. 2024 on active prompting. ...

Both works (Diao et al. 2024 and Dong et al. 2025) are discussed in our related work (§5). They rest on different problem definitions from APEX-DATA and are not comparable:

  • APEX-DATA is designed for automatic prompt optimization for black-box LLMs on broad generation tasks.

  • Diao et al. (2024), Active-Prompt, is not an automatic prompt optimization method, but selects which questions to annotate as few-shot chain-of-thought exemplars. Its problem setup differs from APEX-DATA as the goal is to select data to annotate by humans with no optimization loop.

  • Dong et al. (2025), IPOMP, is designed for prompt optimization for classification tasks with a fixed label set, and requires sentence embeddings and output-label logits.

Neither can be run under our black-box, generative-task protocol without changing its core assumptions. The closest baselines are our Stratified and Diversity Prior selectors (Fig. 3), at 43.5 and 44.9 vs. 52.3 for APEX-DATA. We will expand the related-work paragraph with these specifics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.