- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces APEX-DATA (Automated Prompt Engineering expert with Dynamic Data Selection), a data-centric framework designed to improve the data efficiency of evolutionary prompt optimization under black-box API constraints. Rather than treating the development set as a static benchmark evaluated uniformly, APEX-DATA dynamically stratifies examples into three tiers—Easy, Hard, and Mixed—based on pass/fail outcomes recorded across a sliding lookback window (k=5) of recent prompt variants in the optimization lineage.
The framework concentrates evaluation and generation compute on the volatile Mixed tier through two primary mechanisms:
- Trajectory-Guided Mutation: Conditions prompt edits on recent failures drawn from the Mixed tier (B_{M,0}) rather than intractable Hard errors (B_{H,0}), targeting fixable regressions while maintaining a usage history cache to prevent repetitive cycling.
- Rank-Sensitive Selection: Evaluates candidate prompts against the incumbent prompt using a budget-capped evaluation set D_eval. It prioritizes volatile unknown examples (B_{M,empty}) and employs an annealed anchor-ratio schedule (alpha_t) to balance positive stability checks (B_{M,1}, B_{E,empty}) against negative error corrections (B_{M,0}, B_{H,empty}).
Experiments across three diverse tasks and evaluation regimes—IFBench (programmatic verifier), SimpleQA Verified (exact match), and FACTS Grounding (LLM-as-a-judge)—demonstrate that APEX-DATA outperforms initial prompts by +11.2% on Gemini 2.5 Flash and +6.8% on Gemma 3 27B under a fixed global budget of 5,000 evaluation calls, surpassing baselines such as APO, MIPROv2, TextGrad, and GEPA. Ablation studies validate the utility of finite lookback sizing, Mixed-tier prioritization, and pairwise ranking agreement.
Critical strengths
- Conceptual shift to data-centric optimization: The paper shifts the focus of automated prompt engineering from designing complex meta-prompts and search heuristics to optimizing the informativeness of the evaluation data. Reframing the development set from a static benchmark into a dynamic pool of informative signals directly tackles the primary compute bottleneck in black-box prompt search.
- Principled operationalization of frontiers: Categorizing volatile instances into an "addressable frontier" for mutation (targeting solvable regressions) and a "rank-sensitive frontier" for selection (isolating candidate-discriminating instances) is mathematically straightforward, clean, and intuitive. The adaptive anchor schedule sensibly balances positive preservation against error correction as prompts mature.
- Broad empirical coverage across evaluation regimes: The framework is evaluated across three distinct task archetypes and verification protocols: verifiable programmatic constraints (IFBench), gold-answer factuality (SimpleQA Verified), and model-based evaluation (FACTS Grounding), showing consistent improvements across both proprietary (Gemini 2.5 Flash) and open-weight (Gemma 3 27B) models.
- Informative ablation and analytical experiments: The analyses in Section 4.3 thoroughly validate the core assumptions: Figure 4 verifies that mutating on Hard errors actively degrades performance (-8.2%), Figure 5 confirms substantial improvements in pairwise ranking agreement over uniform random sampling, and Table 4 shows that performance gains are not confounded by prompt length expansion.
Critical weaknesses
- Budget accounting fairness and caching parity: APEX-DATA relies heavily on incremental caching (Algorithms 1 & 2), evaluating only candidate prompts on historically evaluated instances while reusing incumbent results. It is not made clear whether baselines (such as APO, GEPA, and MIPROv2) were given identical caching infrastructure and whether the 5,000-call budget was calculated identically across all methods. If baselines were re-evaluating incumbents or lacking cache reuse, APEX-DATA would effectively enjoy far more search iterations within the same 5,000 evaluation calls. In addition, it is unclear whether the initial full evaluation of P_0 (Algorithm 2, line 2, which requires 500-700 calls) is subtracted from the 5,000 budget.
- Under-specification of cold-start and early-iteration tiering: Tier assignment depends on the last k valid outcomes (R_i). At iteration t=1, every evaluated example has exactly one evaluation (|R_i|=1), meaning its outcome set is either {1} or {0}. Consequently, the Mixed tier is strictly empty (B_{M,s} = empty), D_req is empty, and the pass rate ratio rho_mix in Eq. (10) results in division by zero. The manuscript does not specify how the algorithm bootstraps when t < k or how early cold starts are handled.
- Susceptibility to LLM-judge stochasticity: On FACTS Grounding, binary scores are generated by an LLM-as-a-judge. Because LLM judges exhibit inherent stochasticity and non-determinism, a single random output variation can spuriously push an example from Easy or Hard into the Mixed tier. The paper does not analyze the stability of lineage-based tiering under evaluator noise, nor does it specify whether temperature was set to 0 or multiple judge samples were averaged.
- Inconsistency regarding Hard error sampling: While Section 4.3 and Figure 4 clearly demonstrate that prioritizing Hard instances degrades performance by -8.2% relative to the initial prompt, Eq. (8) and Algorithm 2 line 5 still explicitly sample mutation errors from (B_{M,0} U B_{H,0}) \ U. The paper lacks an ablation analyzing why B_{H,0} is included and what occurs when mutations are restricted strictly to B_{M,0}.
- Lack of statistical significance and run reporting: Table 1 presents results as mean +/- std, but never specifies the number of independent random seeds or runs conducted. Given that several improvements on Gemma 3 27B are moderate (e.g., 93.3 +/- 0.1 vs. GEPA's 91.7 +/- 0.5 on FACTS Grounding), explicit reporting of sample sizes and statistical significance tests (e.g., p-values) is needed to confirm robustness.
Questions for authors
- Budget and Caching Parity: Did all baseline implementations (APO, GEPA, MIPROv2, TextGrad) utilize the exact same incremental caching mechanism as APEX-DATA? Specifically, does the 5,000-call global budget account for the initial full evaluation of P_0 on dataset D, and does an evaluation call refer strictly to task-model inferences or also LLM-judge calls?
- Cold-Start Initialization: How does the framework handle early iterations (t < k), especially iteration t=1 where |R_i|=1 and the Mixed tier is empty? How are D_req, rho_mix, and candidate selection computed when no Mixed instances exist?
- Ablation on Hard Error Inclusion: Given that Figure 4 indicates Hard error prioritization severely degrades performance (-8.2%), why does Eq. (8) sample from both B_{M,0} and B_{H,0}? What proportion of sampled mutation batches ended up containing Hard errors, and have you evaluated performance when restricting mutations strictly to B_{M,0}?
- LLM-Judge Stochasticity: On FACTS Grounding, how did you control for evaluator variance (e.g., judge temperature, majority voting)? Did judge noise cause false positives in the Mixed tier, and how resilient is the lineage window to such classification errors?
- Experimental Seeds and Significance: How many independent runs were performed to compute the mean and standard deviations in Table 1? Are the performance gains over GEPA on Gemma 3 27B statistically significant?
- Meta-Optimizer Overhead: What is the total token consumption and API cost incurred by the meta-optimizer LLM (Critique and Mutate prompts in Listings 4 & 5) and history bookkeeping across the full 5,000 evaluation budget compared to baselines?
2 comments
W5 / Q5: Number of seeds and statistical significance
Table 1 presents results as mean +/- std, but never specifies the number of independent random seeds or runs conducted. ... explicit reporting of sample sizes and statistical significance tests (e.g., p-values) is needed to confirm robustness.
Are the performance gains over GEPA on Gemma 3 27B (e.g., 93.3 +/- 0.1 vs. GEPA's 91.7 +/- 0.5 on FACTS Grounding) statistically significant?
Table 1 reports mean ± std over 3 independent runs per cell. We will state this in the caption and refine the claim to be more precise: "APEX-DATA performs better in most settings and is comparable to the strongest baseline in the others." No single baseline is consistently best. The strongest baseline changes across cells (APO, TextGrad, GEPA).
Welch's t-test of APEX-DATA against every baseline in every cell of Table 1: APEX-DATA's mean is at or above every baseline in all 24 comparisons, and significantly better in 18 of 24 comparisons (one-sided, p < 0.05; 16 of 24 two-sided), and never worse.. On Gemma FACTS, the gain over GEPA is significant (p = 0.014).
Q6: Meta-optimizer overhead
What is the total token consumption and API cost incurred by the meta-optimizer LLM?
APEX-DATA's optimizer overhead is small, 0.1M to 0.5M tokens across models and benchmarks, at most 6% of its total token cost. For comparison, baselines use up to 2.8M (APO), 1.9M (MIPROv2), and 0.9M (TextGrad), with only GEPA lower (up to 0.3M).
We thank the reviewer for the detailed and constructive review. In response, we show that the budget and caching accounting is fair, specify the cold-start fallback, correct Eq. 8, measure the judge's reliability, and report run counts and significance tests.
W1 / Q1: Budget accounting and caching parity
The comparison is fair: every method is charged for the same unit under the same 5,000-call budget, and every baseline runs with its official implementation, including its own caching.
Call budget: one task-model inference plus its scoring (including judge calls), defined the same way for all methods.
P0 evaluation: the initial full evaluation of P0 is counted within the 5,000 budget. Cached retrievals are not counted, for any method.
Baselines and caching:
Incremental evaluation is a design choice and part of APEX-DATA's contribution. It pays off because APEX-DATA's data selection repeatedly targets the same informative instances, so the current best prompt's results can be reused and only the new candidate needs inference. With random sampling, as in the baselines, there is little to reuse.
W2 / Q2: Cold start
Thank you for catching this missing detail. We will add the rule to Alg. 2 and §3.1.
W3 / Q4: LLM-judge stochasticity on FACTS Grounding
Judge protocol: We follow the official FACTS Grounding judge templates and two-stage pipeline (Jacovi et al., 2025), whose templates were selected against human-annotated held-out examples. Each item receives multiple judge calls, including a quality filter, and grounding check sentence by sentence. The item passes only if all checks pass. The judge (Gemini 2.5 Flash, temperature 0) is the same for every method.
Measured reliability: We re-judged fixed responses on all 200 test items three times each. Verdicts were identical across all three repeats for 98% under the optimized prompt. Judge noise is therefore small.
W4 / Q3: Eq. 8 Clarification
We thank the reviewer for catching this inconsistency. As stated in the paper text (line 216, Fig. 4), our method samples mutation errors in priority order: unvisited Mixed failures first, and Hard failures only as a fallback when no unvisited Mixed failures remain. Eq. 8 and Alg. 2 did not reflect this priority, and we will correct both.
Hard errors are rare in mutation batches: across runs, ~10% of mutation batches contain any Hard item, mostly at cold start, when the Mixed tier is empty by construction.
The effect of each tier is already in Fig. 4. Prioritizing Mixed gives +13.8, All +4.4, Easy −1.1 and Hard −8.2. The −8.2% corresponds to prioritizing Hard instances, a different regime from using them as a rare fallback.