acceptodds
Under review as a conference paper at ICLR 2027

GRAFT: Transformation-Level Credit Assignment over Prompt Derivation Graphs

Abstract

Despite rapid advances in large language model (LLM) capabilities, effective prompting remains essential for eliciting their full potential, particularly in compound systems governed by multiple interacting instructions. Automatic prompt optimization (APO) automates this process, yet existing methods organize search around prompt candidates and underuse evidence about the transformations that produce them, leading to inefficient exploration under limited budgets. We introduce Graph-based CRedit Assignment For Prompt Transformations (GRAFT), built on the insight that a proposal's evaluation provides evidence not only about the resulting prompt but also about the realized transformation, making the transformation the natural unit of search and credit assignment. GRAFT records optimization as a multi-parent prompt derivation graph and maintains Monte Carlo statistics on transformation edges, allowing accumulated evidence to guide subsequent search toward productive transformations. To capture different notions of transformation utility, we design two reward formulations for GRAFT: mean reward measures average performance, while coverage reward measures how frequently a transformation's output reaches the current per-instance frontier. Across six benchmarks and two open-source LLMs, GRAFT achieves the strongest aggregate performance; on Qwen3-8B, it outperforms GEPA, the strongest baseline, by 3.14 points while using 4.50% fewer rollouts and 20.0% fewer tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Prompt Optimizatoin, Agentic Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_13W8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GRAFT, a graph-based automatic prompt optimization (APO) framework that makes realized prompt transformations (including multi-parent crossovers) the fundamental unit of search and credit assignment. GRAFT maintains a prompt derivation graph with Monte Carlo statistics on transformation edges and uses edge-conditioned UCB with progressive widening, adaptive operator selection (mutation vs. crossover), and learned partner selection. It also proposes two transformation-level rewards—mean and coverage—that respectively optimize average performance and per-instance frontier coverage, achieving consistent gains over strong baselines across six benchmarks and two task models under matched budgets.

Critical strengths

1.The shift from candidate-centric to transformation-centric search and credit assignment is a clear conceptual advance; multi-parent hyperedges retain full derivation context and enable persistent, reusable statistics for future decisions.

2.The integration of edge-conditioned UCB, progressive widening, adaptive operator selection with plateau-aware crossover triggers, and principled partner selection is well-designed and cohesive.

3.The coverage reward formulates a complementary utility notion (per-instance frontier attainment), which is well-motivated in sparse and heterogeneous evaluation regimes and distinct from candidate-level Pareto selection used in prior wor

Critical weaknesses

1.Ablations are reported on two tasks only; it would be valuable to show broader coverage (especially where crossover contributes vs. not) and to quantify crossover invocation rates and partner diversity by task.

2.Token accounting appears to focus on task-model tokens; it is unclear whether reflection-model tokens are included in Table 2, making cost comparisons potentially incomplete.

3.Sensitivity analyses for key hyperparameters (UCB/PW coefficients, plateau bonus, partner-score weights, coverage tolerance) are omitted; fixed settings across tasks are laudable but mask potential brittleness.

Questions for authors

1.How are QΔ(e, o) (operator advantages) and Q_partner (historical partner utility) initialized, updated, and decayed? Do they back up gated failures and neutral steps, and are there smoothing/prior mechanisms to avoid early overconfidence?

2.The coverage frontier changes over time, making rewards non-stationary. Did you observe instability in UCB estimates or premature convergence? Any empirical mitigation (e.g., reward smoothing, delayed frontier refresh) or theoretical guarantees?

3.Do the token-cost numbers in Table 2 include reflection-model (Gemini) tokens, prompt-construction overhead, and gate minibatches? If not, can you report end-to-end tokens and wall-clock for parity with GEPA?

4 comments

W1: Broader Ablations and Crossover Behavior

Thank you for this question.

(i) Broader ablations. GRAFT's crossover recombines corresponding module instructions from two parent programs. It creates no new module combinations on single-module GPQA, AIME-2025, and LiveBench-Math. We therefore extend the crossover ablation to all three multi-module benchmarks: HotpotQA, IFBench, and PUPA.

The following results use Qwen3-8B with coverage reward and report mean ±\pm SD over three seeds. The ablation disables crossover while keeping the remaining settings and rollout budgets unchanged.

Method / MetricHotpotQAIFBenchPUPA
GRAFT (Full)66.89 ±\pm 0.7035.99 ±\pm 0.4393.24 ±\pm 1.41
GRAFT (w/o crossover)62.11 ±\pm 1.3534.86 ±\pm 0.9491.79 ±\pm 1.16
Δ\Delta (points)+4.78+1.13+1.45

Crossover improves all three multi-module benchmarks, demonstrating its effectiveness. The core transformation-level credit ablation also covers all six Qwen3-8B benchmarks; see our response to W2 of Reviewer MKCu.

(ii) Crossover invocation and partner diversity. Results are averaged over three runs per benchmark. Invocation rate is the fraction of iterations that construct and gate-evaluate a crossover child; partner diversity counts distinct partner edges per run; pair reuse measures repeated selections of the same ordered source-partner pair.

BenchmarkCrossover invocation rate (%)Unique partner edges/runPair reuse (%)
HotpotQA38.59.358.7
IFBench31.84.756.0
PUPA13.63.347.4

These results show that GRAFT explores multiple partner edges while revisiting selected source-partner pairs. HotpotQA exhibits the highest crossover usage and partner diversity, together with the largest performance gain from crossover (+4.78 points).

We will include the expanded ablations and crossover behavior statistics in the revision.

W2 & Q3: End-to-End Optimization Cost

Thank you for requesting this clarification. Table 2 includes task-model tokens spent on reflection-trace collection, gate minibatches, and full DvalD_{\mathrm{val}} evaluations, including costs incurred by rejected proposals. Reflection-model tokens were excluded from Table 2; we add them and total optimization time below. LLM-based prompt construction counts toward reflection-model tokens; local prompt assembly counts toward wall-clock time.

Cost (avg.)Qwen3-8B (GEPA)Qwen3-8B (GRAFT)Gemma3-12B-IT (GEPA)Gemma3-12B-IT (GRAFT)
Task-model tokens (M)10.88.69.37.9
Reflection-model tokens (M)0.2920.2570.4250.414
Total optimization wall-clock time (h)3.532.951.801.55

The table reports the six-benchmark average cost per run for GEPA and GRAFT-coverage. Including reflection-model tokens, GRAFT maintains lower total token usage and shorter optimization time than GEPA on both task models. We will clarify the accounting scope and include this expanded cost breakdown in the revision.

W3: Hyperparameter Robustness

Thank you for this suggestion. Our sensitivity analysis shows that GRAFT maintains strong performance across variations in its key search hyperparameters.

We vary the search hyperparameters listed below on Qwen3-8B across HotpotQA, IFBench, and PUPA. Each sweep changes the indicated parameter or weight configuration while keeping the remaining settings, data splits, and rollout budgets fixed. Partner-weight sweeps vary their relative contributions. Ranges span the default configuration and tested alternatives, using means over three seeds.

ParameterDefaultTested valuesHotpotQAIFBenchPUPA
Default configurationTable 5—66.8935.9993.24
Edge UCB coefficient cc1.4{1.0,1.4,2.0}\{1.0, 1.4, 2.0\}66.00–66.8935.54–35.9992.68–93.35
Operator UCB coefficient copc_{\mathrm{op}}1.0{0.5,1.0,1.5}\{0.5, 1.0, 1.5\}66.11–66.8935.60–35.9992.81–93.24
PW coefficient kk0.7{0.5,0.7,1.0}\{0.5, 0.7, 1.0\}65.89–66.8935.48–35.9992.54–93.24
PW exponent α\alpha0.5{0.3,0.5,0.7}\{0.3, 0.5, 0.7\}65.78–66.8935.43–35.9992.60–93.31
Plateau bonus η\eta2.0{1.0,2.0,3.0}\{1.0, 2.0, 3.0\}66.00–66.8935.60–36.0592.75–93.24
Partner weights (λq,λs,λc,λu,λp)(\lambda_q,\lambda_s,\lambda_c,\lambda_u,\lambda_p)(0.30, 0.30, 0.20, 0.05, 0.05)Equal (0.20×5)(0.20\times 5); Comp-heavy (0.20,0.20,0.40,0.10,0.10)(0.20, 0.20, 0.40, 0.10, 0.10)65.89–66.8935.49–35.9992.63–93.24
Coverage tolerance ϵ\epsilon10−910^{-9}{10−9,0.05,0.25}\{10^{-9}, 0.05, 0.25\}66.89–66.8935.54–35.9992.88–93.62

Across all tested settings, the largest performance drop from the default is only 1.11 points on HotpotQA, 0.56 on IFBench, and 0.70 on PUPA. This robustness across search-allocation parameters and partner weights supports GRAFT's shared configuration across tasks.

Q1: Updating Operator and Partner Statistics

Thank you for requesting these details. QΔ(e,o)Q_\Delta(e,o) and Qpartner(ep)Q_{\mathrm{partner}}(e_p) track immediate score improvements, separately from the downstream edge value Q(e)Q(e).

(i) Initialization and updates. Both estimates and their respective counts are initialized to zero. After evaluating a proposal, each relevant estimate qq and its own count nn are updated as

n←n+1,q←q+δ−qn.n \leftarrow n+1,\qquad q \leftarrow q+\frac{\delta-q}{n}.

Here, δ\delta is the selection-split improvement when available, otherwise the gate improvement averaged over gate examples. Mutation compares the child with its parent; crossover compares it with the better of its two parents on the same evaluation set. Every evaluated proposal updates the source's operator statistics; a realized crossover also updates the selected partner's statistics.

(ii) Failed and neutral proposals. Gate failures contribute their normalized gate deltas to these local statistics, while leaving downstream Q(e)Q(e) unchanged. Fully evaluated proposals contribute their selection-split deltas regardless of graph admission, so rejected regressions and neutral steps are also recorded. If crossover falls back before forming a child, only its operator count is incremented, with its advantage estimate unchanged; no partner outcome is recorded.

(iii) History and exploration. We use incremental averaging with a 1/n1/n update step, without time-based decay or an additional Bayesian prior. The operator UCB bonus (Eq. 5) and partner exploration term (Eq. 6) encourage under-explored choices. Historical partner utility is one component of the partner score, with default weight λp=0.05\lambda_p=0.05. Recency is handled separately by the plateau detector's window of five full-evaluation deltas.

Q2: Handling Non-stationary Coverage Rewards

Thank you for this question. GRAFT explicitly handles the changing coverage frontier by refreshing both transformation rewards and historical edge statistics, as described in Section 3.4.

(i) Credit under the current frontier. Each evaluated artifact retains its per-instance score vector. When the frontier advances, GRAFT recomputes bib_i, reevaluates coverage rewards from the stored scores, and refreshes the affected running value estimates, as described in Section 3.4. This keeps the credit used by UCB aligned with the current frontier without additional model evaluations.

(ii) Empirical robustness. The sensitivity results in W3 show limited performance variation across the tested UCB/PW settings and coverage tolerances on HotpotQA, IFBench, and PUPA, supporting robust performance under the evaluated budgets.

(iii) Structured reward drift. With fixed score vectors and a growing accepted set, each per-instance frontier value is non-decreasing. Consequently, the coverage reward of a fixed transformation is bounded in [0,1][0,1] and can only stay constant or decrease as the frontier advances. The refresh keeps historical credit aligned with this moving reference. We do not claim a formal UCB regret bound or global convergence guarantee.

Dear Reviewer 13W8,

Thank you again for your constructive suggestions. We extended the crossover ablation to all three multi-module benchmarks, showing gains on each, and reported crossover invocation rates and partner diversity. We also added sensitivity analyses for the key search hyperparameters, with strong performance across the tested settings. Our response further clarifies operator and partner updates, coverage-statistics refresh, and the cost breakdown including reflection-model tokens and total optimization time.

We would greatly appreciate it if you could let us know whether these additions address your concerns.

Best regards, The Authors

Dear Reviewer 13W8,

Thank you once more for your time and valuable comments. We would be grateful to know whether our response has resolved your concerns, or whether there are any points we could clarify further.

Best regards, The Authors

Reviewer_MKCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GRAFT, a graph-structured search method for automatic prompt optimization. The key idea is to treat the transformations that generate candidate prompts during prompt optimization as the basic units of credit assignment. GRAFT uses a prompt derivation graph to record operations such as mutation and crossover, and leverages edge-level statistics to guide subsequent search. The authors further design mean reward and coverage reward, and conduct experiments on six benchmarks using two open-source LLMs. The results show that GRAFT achieves better overall performance than baselines such as GEPA, while reducing rollout and token costs.

Critical strengths

The experimental evaluation is relatively comprehensive. The authors conduct experiments on six benchmarks covering diverse tasks, including reasoning, instruction following, privacy-aware delegation, and mathematics. In addition, the experiments use both Qwen3-8B and Gemma3-12B-IT, and report results across multiple random seeds as well as optimization costs.

Critical weaknesses
  1. The fundamental distinction between GRAFT’s transformation-level credit and PromptAgent’s MCTS, HAPO’s context-conditioned UCB, and DGA2D’s transition-level credit remains insufficiently clear.

  2. The artifact-level credit ablation, which directly targets the core contribution, is conducted on only two benchmarks and is therefore insufficient to establish the general effectiveness of transformation-level credit.

  3. Some performance gains are small relative to the reported variance. For example, on PUPA, GRAFT(avg) achieves 95.50±1.09 compared with 95.39±1.55 for GEPA, making the statistical significance of the improvement unclear.

  4. The definition of token usage in Table 2 is insufficiently specified, and it is unclear whether it fully includes reflection calls, gate evaluations, rejected proposals, and other optimization costs.

  5. Some references contain incorrect, outdated, or inconsistent bibliographic information; for example, the publication year of MathArena (MathArena: Evaluating LLMs on Uncontaminated Math Competitions) is incorrect, which affects the overall rigor of the manuscript.

Questions for authors

See weaknesses.

4 comments

W1: Distinction from Related Search Methods

Thank you for requesting this clarification. GRAFT assigns persistent search credit to each realized, potentially multi-parent prompt derivation. Each edge records the parent set, operator, resulting artifact, and source/partner context (Eq. 2), and accumulates downstream search returns.

(i) PromptAgent: from single-parent trees to multi-parent derivations. PromptAgent maintains state–action values and backs up future rewards in a prompt tree. GRAFT explicitly represents crossover as a joint transformation involving both parents. It also separates artifact identity from derivation-specific credit: different transformations can reach the same deduplicated artifact while retaining independent N(e)N(e) and Q(e)Q(e). This preserves the distinct search contexts in which an otherwise identical prompt was produced.

(ii) HAPO: operator preferences versus realized-transformation values. HAPO applies UCB to semantic-unit/edit-operator pairs, using immediate validation improvement as reward. GRAFT distinguishes such local operator preferences, represented by QΔ(e,o)Q_\Delta(e,o), from Q(e)Q(e), which accumulates downstream returns for a specific realized derivation. Subsequent exploration therefore updates the value of that derivation while retaining its parent and lineage context.

(iii) DGA2D: pipeline transitions versus prompt derivations. DGA2D assigns credit to transitions between consecutive code implementations within an algorithmic pipeline, using the completed algorithm's performance to guide composition. GRAFT's edges describe how complete prompt programs are generated through mutation or crossover. Its statistics capture optimization history, including the joint parent set, and guide further search through derivation-level traversal, adaptive operator selection, and partner selection.

We will make these distinctions explicit in Section 2.

W2: Broader Ablation of Transformation-Level Credit

Thank you for suggesting broader validation. We extend the artifact-level-credit ablation reported in Table 3 from two to all six benchmarks on Qwen3-8B, following the configuration described in Appendix A.3.

Both variants use the same seed programs, data splits, task and reflection models, random seeds, and rollout caps, including gate evaluations. Per-benchmark results are mean ±\pm SD over 3 seeds; Aggregate is the unweighted mean across the six benchmarks. GRAFT uses coverage reward, and Δ\Delta denotes GRAFT minus the artifact-level-credit variant.

Method / MetricHotpotQAIFBenchPUPAGPQAAIME-25LB-MathAggregate
GRAFT66.89 ±\pm 0.7035.99 ±\pm 0.4393.24 ±\pm 1.4144.60 ±\pm 3.1258.00 ±\pm 2.8341.22 ±\pm 1.6356.66
GRAFT with artifact-level credit65.33 ±\pm 0.5134.88 ±\pm 0.6791.98 ±\pm 0.9636.80 ±\pm 3.0754.67 ±\pm 1.3436.17 ±\pm 0.4153.31
Δ\Delta (points)+1.56+1.11+1.26+7.80+3.33+5.05+3.35

GRAFT achieves higher mean performance on all six benchmarks, improving the aggregate by 3.35 points. The gains extend beyond the original HotpotQA and IFBench ablations, reaching +7.80 points on GPQA, +3.33 on AIME-25, and +5.05 on LB-Math. Since the two variants differ only in credit assignment, these expanded results support the benefit of transformation-level credit across diverse tasks on Qwen3-8B. We will include the complete ablation results in the revision.

W3: Statistical Reliability and Scope of Improvements

Thank you for raising this point. We clarify the interpretation of the PUPA result and the evidence supporting our main conclusions below.

(i) Comparable performance on PUPA. On Qwen3-8B/PUPA, GRAFT-mean achieves 95.50±1.0995.50 \pm 1.09 versus 95.39±1.5595.39 \pm 1.55 for GEPA. The 0.11-point difference is small relative to the reported seed variation. We interpret this setting as comparable observed performance.

(ii) Overall gains across tasks and models. Our overall-performance claim is based on the fixed coverage configuration, which achieves higher mean scores than GEPA in 9 of 12 model-benchmark settings and improves the aggregate by 3.15 and 1.36 points on Qwen3-8B and Gemma3-12B-IT, respectively. Larger task-level gains include +7.10 and +5.89 points on GPQA. This broader pattern supports the overall improvement claim.

(iii) Evidence for transformation-level credit. The controlled ablation in W2 directly examines the proposed credit assignment. Holding the reward and search policies fixed, transformation-level credit improves mean performance on all six Qwen3-8B benchmarks and raises the aggregate by 3.35 points over artifact-level credit. This comparison isolates the effect of credit assignment and supports its contribution across diverse tasks.

We will make these distinctions explicit in the revised discussion of the main results and ablations.

W4: Clarifying Token Accounting

Thank you for requesting this clarification. Table 2 includes task-model tokens spent on reflection-trace collection, gate minibatches, and full DvalD_{\mathrm{val}} evaluations, including costs incurred by rejected proposals. Tokens consumed by reflection-model calls were not included in the original Table 2; we report them separately below, alongside total optimization wall-clock time.

Cost (avg.)Qwen3-8B (GEPA)Qwen3-8B (GRAFT)Gemma3-12B-IT (GEPA)Gemma3-12B-IT (GRAFT)
Task-model tokens (M)10.88.69.37.9
Reflection-model tokens (M)0.2920.2570.4250.414
Total optimization wall-clock time (h)3.532.951.801.55

The table reports the six-benchmark average cost per run for GEPA and GRAFT-coverage. Including reflection-model tokens, GRAFT maintains lower total token usage and shorter optimization time than GEPA on both task models.

W5: Reference Corrections

We apologize for the bibliographic errors. We have rechecked all references for accuracy and consistency. The correct MathArena citation is

Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on Uncontaminated Math Competitions. Advances in Neural Information Processing Systems, 38, 2025.

We will correct this entry and the other identified bibliographic inconsistencies in the revised manuscript.

Dear Reviewer MKCu,

Thank you again for your constructive suggestions. We clarified GRAFT’s distinctions from PromptAgent, HAPO, and DGA2D, and expanded the credit ablation to all six benchmarks on Qwen3-8B. Transformation-level credit improves performance on every benchmark, with an aggregate gain of 3.35 points over artifact-level credit. We also clarified the interpretation of seed variation, reported reflection-model tokens and optimization time, and provided corrections to the references, including MathArena.

We would greatly appreciate it if you could let us know whether these additions address your concerns.

Best regards, The Authors

Dear Reviewer MKCu,

Thank you once more for your time and valuable comments. We would be grateful to know whether our response has resolved your concerns, or whether there are any points we could clarify further.

Best regards, The Authors

Reviewer_Vi1SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: GRAFT is a prompt optimizer for compound LLM systems whose organizing claim is that the unit of search and credit should be the transformation, not the candidate. Existing APO methods store history around prompt candidates or coarse operator preferences, so evidence about "which edit worked, from which parent(s), in which context" never becomes reusable search statistics. GRAFT records the run as a prompt derivation graph: artifacts are vertices, realized mutations and module-level crossovers are edges, and crossovers are directed hyperedges with two parents. Each edge carries Monte Carlo statistics , where Q(e) estimates the expected downstream return of that transformation context rather than the quality of the child alone. Those edge statistics then drive three decisions: which edge to descend/expand , which operator to apply (mutation vs crossover, using a parent-relative advantage Q_delta plus a plateau bonus that switches to crossover when local editing stops paying), and which partner to cross over with (a five-term score over frontier/top-K accepted edges: downstream promise, individual strength, per-instance complementarity, under-exploration, historical partner utility, with module-aware recombination and a similarity cutoff). Proposals are screened on a small gate minibatch, and only candidates with delta_val >= -0.005 are admitted, which keeps near-neutral stepping stones while excluding regressions. Two reward formulations define transformation utility: mean reward (average child score) and coverage reward (fraction of selection instances on which the child reaches the current per-instance frontier), the latter motivated by the fact that candidates in this space are routinely complementary.

Contributions

  1. A transformation-centric APO framework: a multi-parent prompt derivation graph in which edges, not artifacts, are the objects of credit assignment, with exact deduplication by prompt hash so distinct transformations ending at the same artifact keep separate statistics.
  2. Two transformation-level reward formulations, mean and coverage, intended to cover tasks whose gains are broadly shared versus concentrated on complementary instance subsets, plus the corresponding backup rule over the descent path, source edge, and new edge.
  3. Adaptive control of crossover: learned operator timing (plateau-triggered) and learned partner selection via an explicit complementarity term, rather than a fixed crossover rate.
  4. Empirical claim: strongest aggregate result on six benchmarks and two open-source task models at matched or lower rollout/token cost than GEPA, with ablations attributing the gain to transformation-level credit and adaptive crossover.

Validation Setup: HotpotQA, IFBench, PUPA, GPQA-Diamond, AIME-2025, LiveBench-Math; Qwen3-8B and Gemma3-12B-IT as task models; Gemini 3.1 Pro as the fixed reflection model; three seeds; unweighted mean over the six test scores; budgets matched to MIPROv2's metric-call caps per benchmark, with gate evaluations counted against the budget. Baselines: seed program, MIPROv2, PromptAgent, GEPA, GEPA+Merge.

Results: GRAFT-coverage reaches 56.65 aggregate on Qwen3-8B (+3.14 over GEPA, which is +3.14 -> 53.51) and 49.88 on Gemma3-12B-IT (+1.36); best on all six tasks for Qwen3 and four of six for Gemma. Coverage beats mean on 10 of 12 model-benchmark pairs (+1.61 / +2.03 aggregate), with mean stronger on IFBench and PUPA under Qwen3. Cost: 4.50% / 3.31% fewer rollouts and 20.0% / 14.6% fewer task-model tokens than GEPA on average. Ablations (Qwen3 only, coverage reward, two benchmarks): artifact-level credit, fixed-rate crossover, random partner, and no-crossover all degrade performance, with removing crossover costing the most on HotpotQA (-4.72). Appendix adds optimization curves and derivation-graph visualizations for GRAFT versus PromptAgent and GEPA+Merge, plus full prompt programs and hyperparameter/split tables.

Critical strengths

First, the paper correctly identifies a real defect in the search state of existing APO methods and fixes it with an abstraction that is both general and cheap to adopt. Current optimizers retain candidate quality, program parameters, or a proposal policy; what they discard is precisely the evidence "this edit, applied to this parent set, in this context, was productive," and that evidence is what the paper makes persistent and actionable. The mechanism is not a decorative wrapper: mutation is literally GEPA's reflective update, so edge-level statistics can be layered onto an existing optimizer without replacing it, and the graph does real work — it distinguishes transformations that converge on the same deduplicated artifact, which candidate-level bookkeeping provably cannot. Why this is consequential to me: it is the reason I am willing to accept an incremental-sounding paper. This contribution is order-independent of the empirical result; if the numbers were regenerated with more seeds or other models, the reformulation and the mechanism would still be there, and the community can build on it. It is also the part of the paper that the ablation, however thin, actually addresses (artifact-level credit loses 1.56/1.11 under otherwise identical search), so the claim is at least pointed in a testable direction.

Second, the coverage reward is a substantive claim about what makes a transformation valuable, and the paper tests it rather than merely asserting it. Reframing utility as per-instance frontier attainment on edges — not average child quality — encodes the correct structural fact about this search space, namely that candidates are complementary and an unremarkable mean can hide the only candidate that solves a hard subset. The evidence is the strongest empirical part of the paper: coverage beats mean on 10 of 12 model-benchmark pairs with consistent gains concentrated exactly on the sparse, answer-level benchmarks where complementarity should matter, and the authors report the counter-cases where mean wins (IFBench, PUPA under Qwen3) with a mechanistic explanation instead of burying them. Why this is consequential: it partially rescues the paper from my strongest objection. I noted that mean-reward GRAFT loses to GEPA on Gemma, which undercuts the graph machinery per se; but that same fact is what makes the reward finding interesting rather than incidental — the utility definition, not the data structure, carries most of the gain, and the paper is honest about reporting it. A paper that reports the configuration where its own architecture underperforms, and explains the pattern, is giving me evidence I can use even if I discount the headline delta.

Critical weaknesses

Weakness 1 — the contribution is not isolated by any experiment the paper runs.

Swap only the utility definition and keep everything else: GRAFT-mean trails plain GEPA on the Gemma aggregate (47.85 vs 48.52) and by 5.5 points on Gemma/HotpotQA (56.84 vs 62.34). Since GEPA is the baseline being extended, that comparison says the derivation graph, edge statistics, and adaptive crossover do not beat the baseline on their own; the gain lives in the coverage reward plus the stricter gate. The control that would settle this is one row — GEPA with GRAFT's gate and GRAFT's per-instance frontier reward, no edge statistics — and it is absent. The only direct evidence for the edge abstraction is the artifact-level-credit ablation, a 1.56-point drop against ±0.70 seed noise, on one model and two of six benchmarks. The mechanism is also underspecified exactly where the claim lives: the text says a rejected proposal increments visits without changing values, while Algorithm 1's rejection branch backs the source edge's own reward through the path, so reward from no realized transformation enters the Q values that steer the search. Consequential because it attacks the one strength I was willing to accept the paper on. My accept case was that the reformulation is general, adoptable, and independent of the headline number; if its causal effect is unmeasured and its credit rule is implemented inconsistently, the defensible description of the contribution shrinks to a reward formulation and a screening gate drawn on a graph — narrower than the title, abstract, and contribution list claim.

Weakness 2 — the claims are pitched far above the statistical and economic evidence.

The headline is one unweighted six-task mean over three seeds, no significance testing, overlapping intervals on the decisive Gemma comparison (49.88 ± 1.19 vs 48.52 ± 1.15), two of six Gemma tasks lost, AIME resting on 30 problems, and an ablation reported as bare means but described as showing the components are "critical." Meanwhile the paper's stated motivation is budget efficiency, yet the cost table counts only task-model rollouts and tokens while the reflection model is a proprietary frontier system called uncached every iteration and the coverage bookkeeping overhead conceded in Limitations is never measured; the 4.50% rollout saving is an artifact of caps inherited from MIPROv2 rather than a property of the method. Consequential because with effects of 1–3 points on benchmarks whose seed variation is the same order, and the efficiency advantage unmeasured in the currency a practitioner pays, I cannot separate a real but modest improvement from a favorable seed draw at the rigor this venue should demand — and the two claims most likely to drive adoption are precisely the two least supported.

Questions for authors
  1. Do you have the control that isolates your contribution: GEPA's search with your gate and your per-instance frontier reward, but statistics attached to artifacts or to nothing, i.e. no edge-level UCB, no adaptive operator choice, no partner selection? One row on Qwen3 and Gemma would settle whether the gain is transformation-level credit or the reward/gate combination. If it does not already exist, running it in the rebuttal period is the single most valuable thing you can do; if it shows the graph adds little, I would rather revise my assessment than have the paper claim otherwise.

  2. Why does GRAFT-mean lose to GEPA on Gemma (47.85 vs 48.52 aggregate; 56.84 vs 62.34 on HotpotQA) when the data structure is held fixed and only the utility definition changes? Is there a mechanism-specific cost under a dense mean signal — admission tolerance pruning useful stepping stones, widening quotas misallocating budget, or deduplication collapsing lineages? An explanation consistent with your Section 4.2 account of when mean should win would substantially raise my confidence in the framework; an unexplained loss leaves the graph's value in doubt.

  3. Which of the two is what you implemented: the text's rule that a proposal failing to reach selection-split evaluation increments visits while leaving values unchanged, or Algorithm 1's rejection branch, which backs the source edge's own reward through the path? If the former, the inconsistency is editorial and I discount it. If the latter, the Q values steering UCB contain reward from transformations that were not realized, and I need to know why that is the right estimator rather than a bug.

  4. How much of the coverage gain is attributable to the gate rather than to credit assignment? Please report gate acceptance and rejection rates per benchmark, and — importantly — whether GEPA applies any comparable pre-screen before spending a full selection-split evaluation. GEPA as I understand it does not, which would make "fewer rollouts and 20% fewer tokens" partly a property of screening rather than of edge statistics.

Suggestions for future work (explicitly not required for my assessment, and I will not penalize their absence): validating the mechanism with a weaker or open reflection model and larger task models; more than four modules; richer transformation operators than mutation and module crossover; sensitivity analysis over the partner weights, the plateau threshold, and the widening parameters; quantifying the bookkeeping overhead you flag in Limitations; and transferring edge statistics across related tasks, which is the most interesting direction your framework opens.

4 comments

W1 & Q1: Isolating Transformation-Level Credit

Thank you for this question. Our controls show gains beyond coverage and gating, including the contribution of transformation-level credit.

(i) GEPA+Gate+Coverage. This control matches GRAFT's independent gate, admission tolerance, and per-instance frontier reward. It samples parents in proportion to their current coverage rewards, retaining GEPA's mutation-only search without edge statistics, adaptive operator selection, or partner selection.

(ii) GRAFT with artifact-level credit. We extend the Table 3 credit ablation to all six benchmarks and both task models. This variant replaces edge-specific N(e)N(e) and Q(e)Q(e) with statistics shared by transformations reaching the same artifact. The reward, gate, admission rules, operator selection, and partner selection otherwise remain unchanged. Qwen3-8B per-benchmark results appear in our response to W2 of Reviewer MKCu.

All comparisons match data splits, task and reflection models, and rollout caps, including gate evaluations. Scores are unweighted six-benchmark averages of three-seed means; GRAFT (Ours) uses coverage reward.

MethodQwen3-8BGemma3-12B-IT
GEPA+Gate+Coverage54.4848.86
GRAFT with artifact-level credit53.3149.04
GRAFT (Ours)56.6649.88

GRAFT improves over the matched GEPA control by 2.18 and 1.02 points on Qwen3-8B and Gemma3-12B-IT, respectively. Sharing credit at the artifact level reduces the aggregate by 3.35 and 0.84 points, respectively, with lower mean performance on all six Qwen3-8B benchmarks. These comparisons establish the benefits of GRAFT's search framework and transformation-specific credit, respectively. We will include these controls in the revision.

Q2: Mean vs. Coverage on Gemma

Thank you for this question. Coverage improves HotpotQA on both task models, while the ranking relative to GEPA also depends on the search configuration and task model.

(i) Reward suitability on HotpotQA. HotpotQA combines heterogeneous multi-hop reasoning with binary answer-level exact match. Holding the GRAFT search framework fixed, coverage improves over mean on both Qwen3-8B (66.8966.89 vs. 64.6764.67) and Gemma3-12B-IT (60.4460.44 vs. 56.8456.84). Section 4.2 motivates mean for graded, broadly shared improvements and defines coverage through per-instance frontier attainment. This isolates reward choice within GRAFT; comparisons with GEPA additionally change screening and search policy.

(ii) Model-dependent performance relative to GEPA. GRAFT-mean has higher mean scores than GEPA on all six Qwen3-8B benchmarks (55.0455.04 vs. 53.5153.51 aggregate). On Gemma3-12B-IT, its aggregate deficit is concentrated on HotpotQA (56.8456.84 vs. 62.3462.34); across the other five benchmarks, its average is higher (46.0546.05 vs. 45.7645.76).

A plausible explanation is model-dependent sensitivity to multi-module search. GRAFT-mean combines mean-return edge traversal, an independent gate, and adaptive crossover, while GEPA uses per-instance Pareto parent selection. GEPA+Merge shows a similar pattern: enabling merge raises HotpotQA from 62.8362.83 to 64.8364.83 on Qwen3-8B, but lowers it from 62.3462.34 to 54.6654.66 on Gemma3-12B-IT. This reversal suggests that the effectiveness of recombination depends on the task model and can contribute to the different rankings against GEPA. The matched controls in Q1 separately assess the search-framework and credit-assignment contributions.

W1 & Q3: Credit Assignment for Rejected Proposals

Thank you for highlighting this ambiguity. Both rules are implemented, but apply to different stages of proposal evaluation.

(i) Gate rejection (Algorithm 1, lines 25-26). A gate-rejected proposal never reaches selection-split (DvalD_{\mathrm{val}}) evaluation. As stated in Section 3.4, we increment visits N(e)N(e) along the traversed path while leaving all edge values Q(e)Q(e) unchanged, using the normalized gate delta to update operator statistics and, for crossover, partner statistics.

(ii) Graph-admission rejection after full evaluation (Algorithm 1, lines 21-24). When a proposal passes the gate and is evaluated on DvalD_{\mathrm{val}} but fails the admission condition (Δval<−ϵtree\Delta_{\mathrm{val}} < -\epsilon_{\mathrm{tree}}), Algorithm 1 backs up Rρ(es)R_\rho(e_s). Here, Q(e)Q(e) estimates downstream search return under the admission policy:

  • Because no admissible successor is added, the expansion retains the source artifact. We therefore use its evaluated utility Rρ(es)R_\rho(e_s) as this iteration's return.
  • This backup moves Q(es)Q(e_s) toward the source utility Rρ(es)R_\rho(e_s), leaving it unchanged when the two are equal, while incrementing N(es)N(e_s) to update its UCB exploration term.
  • The negative Δval\Delta_{\mathrm{val}} separately updates operator advantages, partner utility when applicable, and the plateau window, informing subsequent operator selection and plateau detection.

W2: Statistical Reliability and Scope of Improvements

Thank you for raising the question of reliability. The 1.191.19 and 1.151.15 values in Table 1 are the standard deviations for Gemma/HotpotQA, rather than for the six-benchmark aggregates.

With the fixed coverage configuration, GRAFT achieves higher mean performance than GEPA in 9 of 12 model-benchmark settings, with aggregate improvements of 3.15 and 1.36 points on Qwen3-8B and Gemma3-12B-IT. Several task-level gains are substantial in magnitude, including +7.10 and +5.89 points on GPQA for the two models. Together with the matched controls and credit ablation in Q1, these results support the benefit of transformation-level credit in guiding prompt optimization. Excluding AIME, GRAFT-coverage still improves the five-benchmark average over GEPA by 2.93 points on Qwen3-8B and 1.10 points on Gemma3-12B-IT.

Smaller gains or regressions may reflect differences in reward suitability and model-specific responses to prompt transformations. For example, on Qwen3-8B/PUPA, GRAFT-mean reaches 95.50 compared with 93.24 for GRAFT-coverage, consistent with Section 4.2's motivation for mean reward on graded, broadly shared improvements. The model-dependent effects of recombination discussed in Q2 provide another possible source of variation. We will explore adaptive reward selection and search policies that better accommodate these model-task differences in future work.

W2 & Q4: Gate Design and End-to-End Cost

Thank you for these questions. We clarify the pre-screening mechanism in GEPA vs. GRAFT, report per-benchmark gate statistics, and provide the complete end-to-end cost accounting below.

(i) Pre-screening in GEPA vs. GRAFT and gate acceptance rates.

GEPA also performs minibatch pre-screening: in our baseline configuration, a proposed mutation reaches full DvalD_{\mathrm{val}} evaluation only if it improves over its parent on the three reflection examples. GRAFT uses an independently sampled gate minibatch of the same size. The GEPA+Gate+Coverage control in W1 & Q1 compares the two search frameworks under matched screening and reward.

The table below reports gate acceptance rates and conditional graph admission rates among gate-accepted proposals (Δval≥−ϵtree\Delta_{\mathrm{val}} \ge -\epsilon_{\mathrm{tree}}) for GRAFT-coverage. Each entry is averaged over 3 seeds; Avg. is the unweighted mean across the six benchmarks:

Metric (%)HotpotQAIFBenchPUPAGPQAAIME-25LB-MathAvg.
Gate acceptance rate (Qwen3-8B)8.716.217.021.818.928.418.5
Graph admission rate (Qwen3-8B)93.989.546.232.463.617.457.2
Gate acceptance rate (Gemma3-12B-IT)8.927.812.023.88.720.917.0
Graph admission rate (Gemma3-12B-IT)57.690.087.037.145.860.963.1

On average, gate rejection rates are 81.5%81.5\% (Qwen3-8B) and 83.0%83.0\% (Gemma3-12B-IT). Crucially, averaged across benchmarks, 42.8%42.8\% (Qwen3-8B) and 36.9%36.9\% (Gemma3-12B-IT) of gate-accepted proposals still fail graph admission (Δval<−ϵtree\Delta_{\mathrm{val}} < -\epsilon_{\mathrm{tree}}), especially on reasoning and math benchmarks (e.g., GPQA, AIME-25, LB-Math). These rates show that passing the minibatch gate does not guarantee satisfying the full-evaluation admission criterion. The subsequent admission check filters proposals that pass the gate but regress beyond the allowed tolerance on DvalD_{\mathrm{val}}.

(ii) End-to-end cost accounting.

Both GEPA and GRAFT apply minibatch pre-screening, with GRAFT using an independently sampled gate as described above. Table 2 reports task-model rollouts and tokens incurred during optimization, including reflection trace collection, gate evaluations, rejected proposals, and full DvalD_{\mathrm{val}} evaluations. Under the matched rollout caps inherited from MIPROv2, GRAFT uses 20.0%20.0\% fewer task-model tokens on Qwen3-8B and 14.6%14.6\% fewer on Gemma3-12B-IT on average.

Below we summarize the six-benchmark average optimization cost per run for GEPA and GRAFT-coverage, including reflection-model tokens and total wall-clock time.

Cost (avg.)Qwen3-8B (GEPA)Qwen3-8B (GRAFT)Gemma3-12B (GEPA)Gemma3-12B (GRAFT)
Task-model rollouts3,1503,0083,1273,023
Task-model tokens (M)10.88.69.37.9
Reflection-LM tokens (M)0.2920.2570.4250.414
Total optimization wall-clock time (h)3.532.951.801.55

GRAFT also uses fewer reflection-model tokens on both task models: 0.257M versus 0.292M on Qwen3-8B, and 0.414M versus 0.425M on Gemma3-12B-IT. Thus, the token savings persist when reflection-model usage is included. Measured bookkeeping overhead, including graph maintenance and coverage-statistics updates, is below 1s1\text{s} per run and accounts for less than 0.05%0.05\% of total optimization wall-clock time. We will include this complete cost summary in the revision.

Dear Reviewer Vi1S,

Thank you again for your constructive suggestions. We added the GEPA+Gate+Coverage control on both task models and expanded the credit ablation to all six benchmarks. These comparisons demonstrate gains beyond coverage and screening, supporting the contribution of transformation-level credit. We also clarified the update rules for rejected proposals, reported gate acceptance and graph admission rates, and provided cost accounting including reflection-model tokens and total optimization time.

We would greatly appreciate it if you could let us know whether these additions address your concerns.

Best regards, The Authors

Dear Reviewer Vi1S,

Thank you once more for your time and valuable comments. We would be grateful to know whether our response has resolved your concerns, or whether there are any points we could clarify further.

Best regards, The Authors

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is approaching the deadline. Please read through the author's rebuttals and update your reviews. Thank you for your time and work.

Best Regards, AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.