acceptodds
Under review as a conference paper at ICLR 2027

GraphSkillEvo: Evolutionary Optimization of Graph-Structured Agent Skills

Abstract

Skills can improve the performance of Large Language Model (LLM) agents by providing task-specific procedural guidance, while skill optimization further improves their effectiveness through iterative refinement. However, existing skill optimization methods typically represent skills as unstructured natural-language instructions, creating two key challenges: **1)** Unstructured skills often lack explicit workflow-level guidance and contain substantial redundancy, making them difficult for LLMs to execute; **2)** the vast search space of unconstrained natural-language skills makes skill optimization ineffective. To address these challenges, we propose representing skills as **graph-structured natural-language artifacts**. In graph-structured skills, each node represents an execution step together with its operational guidance, while directed edges encode context-dependent transitions between steps. Compared to unstructured skills, graph-structured skills can provide clear workflow-level guidance. Moreover, the proposed graph-structured skill can also facilitate skill optimization. Building on this structured representation, we introduce **GraphSkillEvo**, a population-based evolutionary optimization framework with mutation and crossover operators for graph-structured skills. By maintaining multiple candidate skills and combining effective components, GraphSkillEvo enables broader and more comprehensive exploration of the structured skill space than purely LLM-based iterative self-refinement. Extensive experiments across five agent benchmarks demonstrate that GraphSkillEvo outperforms the strong skill optimization baseline SkillOpt, improving average accuracy by 4.01% on GPT-5.4-nano and 1.76% on GPT-5.4. Our code is available at https://anonymous.4open.science/r/GraphSkillEvo.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill Optimization, Agent Skills, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3F7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper represents agent skills as reusable instruction nodes connected by condition-specific workflows, and optimizes these artifacts using population-based selection and four LLM-driven mutation/crossover operators. It reports improvements over SkillOpt across five benchmarks, two models, and two execution settings, together with representation/operator ablations and transfer from GPT-5.4-nano to GPT-5.4.

Critical strengths
  1. The combination of reusable procedural components and evolutionary optimization is well motivated and potentially useful. The skill format and operator prompts are described in considerable detail.
  2. The empirical improvements are encouraging: GraphSkillEvo has the best mean result in 13 of 14 evaluated settings. The 10.60-percentage-point gain over SkillOpt on SpreadsheetBench with GPT-5.4-nano is particularly notable.
  3. The paper includes meaningful supporting experiments, including mutation/crossover ablations, candidate-count-matched single-trajectory and multi-start variants, optimization-token curves, and ten additional runs on four benchmarks. The reported aggregate optimization-token usage is also lower than SkillOpt's.
Critical weaknesses
  1. The specific contribution beyond existing optimization approaches needs clearer evidence. EvoPrompt and GEPA explore evolutionary prompt optimization, AFlow searches agentic workflows, and Evolutionary Programmer constructs planners through evolutionary operator composition. These studies raise a central question: what additional benefit comes from evolving graph-structured skills? A focused discussion of these connections, together with a comparison against a representative evolutionary prompt optimizer beyond SkillOpt under comparable budgets, would strengthen the novelty claim.

  2. The graph ablation in Table 2 removes applicability conditions and execution order, as acknowledged in Appendix E.4. It therefore does not isolate graph organization from additional procedural information. An information-preserving textual workflow baseline would strengthen the central claim.

  3. Robustness evidence is uneven across settings. The ten-run analysis covers only GPT-5.4-nano without a harness, while improvements on GPT-5.4 are smaller. Please report uncertainty for these results and qualify portability claims, since transfer is tested only within one model family.

Questions for authors

What benefit does GraphSkillEvo provide beyond existing evolutionary prompt optimizers such as GEPA or EvoPrompt under comparable optimization budgets?

Does the improvement persist when a non-graph textual workflow preserves the same applicability conditions, execution order, and node-level instructions? This would help distinguish the benefit of graph organization from additional procedural information.

How consistent are the smaller gains on GPT-5.4 across repeated runs? Please provide run-level variability or confidence intervals for the main comparisons.

Reviewer_dQHzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the optimization or agent skills. Compared with previous skill optimizers that treat skill as unstructured text for optimization, this paper produces another method, which represents a skill as a structured graph, consisting of global guidance, a set of reusable execution steps and workflows that connects these steps under different problem solving situations. The skill is then optimized with evolutionary search methods. The experiments on five benchmarks with GPT-5.4 and GPT-5.4-nano show that the proposed method outperforms several baselines.

Critical strengths
  • Practical problem: the paper studies a useful problem of automatic skill optimization for LLM agents.

  • Simple and effective idea: the proposed method is simple and intuitive. The experiments also show the effectiveness of proposed method.

Critical weaknesses
  • Limited evaluation benchmarks: The experiments are conducted on five simpler benchmarks that does not have complicated states. For example, SearchQA/DocQA/LiveMathemathicianBench are single-turn question QA benchmarks., and the SpreadsheetBench/ALFWorld are limited in the state complexity. This paper claims that graph-structured skills help with state transition during problem solving, which should be reflected on more complex benchmarks such as terminal coding/software engineering/web browsing tasks that do require tracking a wide range of states. It would be beneficial to involve experiments on these benchmarks.

  • Potential unfair comparison: GraphSkilLEvo starts from a population of different skills and uses a different optimization budget, while SkillOpt refines a single intiial skill. The ablation experiment also show that evolutionary search without graph structure does not outperform SkillOpt. It's unclear whether the improvemtns of GraphSkillEvo to SkillOpt results from the structured view of skill or the larger optimization budget.

  • Questionable design of evolutionary operations: several design chioces in the search are not clearly explained. For example, the crossover prompts in appendix seem to receive only two parent skills, without any exeuction feedback such as the scores of agent with the previous skill. How does LLM decide which parts to keep during the crossover? It would be beneficialto involve some example for input and output for these oprations. The ablation experiments wihtout mutation also does not perform sigficianly worse, suggesting that the crossover may not help too much for the performance.

  • Too small search budget: The experiment runs with a limited search budget, containing only 4 skill branches. This budget is too small in the sense of a evolutionary search process. It's unclear whether the search process actually help or it's just a simple best-of-N sampling with some refinement during the sampling process.

  • Limited model coverage: All experimtns conducted with GPT-5.4 and 5.4-nano. It's not clear how the optimization mechanim work for smaller/weaker models.

  • Minor:

    • What's the optimization cost for each dataset? Seems that the paper only reports sum of token cost for all benchmarks.
    • Does the agent actually follow new workflow after two parent skill being integrated together?
Questions for authors

Please check weakness section for questions.

Reviewer_KME3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper represents an agent skill as a graph-structured natural-language artifact: global guidance h_s, a node library V_s (each node = one execution step plus its operational instructions), and an edge set E_s given implicitly by M workflows {(c_m, p_m)}, each pairing an applicability condition with an ordered path through nodes. The artifact is serialized into a three-section markdown document (## Global Guidance, ## Node Lists, ## Task Graphs). On top of this representation the authors propose GraphSkillEvo: population-based evolution (N = 4, T = 5) with four LLM-implemented operators — global-guidance mutation, graph-structure mutation, global-guidance crossover, graph-structure crossover — round-robin operator selection, rank-proportional parent selection (p ∝ 1/(r+N)), and elitist truncation by validation accuracy. Experiments cover 5 benchmarks, 2 models (GPT-5.4 / GPT-5.4-nano), 2 harness configurations (no harness / Codex SDK), baselines of no-skill, human-written, one-shot-LLM-written and SkillOpt-optimized skills, plus representation/operator/population ablations, a nano→5.4 transfer study, a 10-run significance test on one model, and a token-consumption comparison. Two scope facts frame my concerns: all five benchmarks are short-horizon, single-artifact tasks (three of them are single-turn in the no-harness row), and both models run at one fixed reasoning effort ("medium").

Critical strengths

S1. The representation is cleanly specified and the artifacts are legible. Sectioning the document lets the operators be defined by which part may change, which is a real design lever, and the appendix shows the actual optimized artifacts so readers can judge them. Reproducibility effort (code link, all operator prompts, dataset splits, hyperparameters) is above the norm for this literature.

S2. The paper is clearly written and easy to follow; the motivation chain (unstructured skills → weak workflow-level guidance + redundant search space → structured representation + population search) is coherent.

Critical weaknesses

W1. The central claim — graph vs. unstructured at matched information — is never tested, and the authors say so

Section 5.1 builds the "unstructured counterpart" by removing explicit workflow organization "while retaining task-general guidance and node-level instructions", and Appendix E.4 concedes that this "examines the contribution of that explicit workflow guidance, rather than comparing graph and non-graph representations that preserve equivalent workflow information." That is exactly the comparison the paper's contribution depends on. The unstructured control therefore has strictly less information: it loses the ordering and the "use when" routing, i.e. it is precisely the "lengthy unordered bullet list" the paper criticizes in the introduction. Table 2 (−4.52/−2.50/−4.19/−1.35/−0.75 on SearchQA/Spreadsheet/DocVQA/LiveMath/ALFWorld) and the w/o graph structure row of Table 3 (−4.10/−10.12/−8.11) consequently cannot distinguish

  1. "graph organization helps" from
  2. "an ordered procedure with task-type routing helps" (which is not a graph claim at all), from
  3. "longer, more specific guidance helps".

Required controls. (i) A content-matched, length-matched flattening: identical node text, identical "Use when" lines, rendered as Step 1 … Step N under each task type, at the same token budget. (ii) A step-permuted version of that flat SOP, to isolate the contribution of ordering specifically. (iii) A length-matched placebo skill (generic guidance of the same size), to bound the "any extra guidance helps" effect. Without (i)–(iii) the headline representation claim is unsupported.

A related conceptual issue. The "graph" is a document layout, not an executable structure. The agent is asked in prose to "select the graph using its Use when line, then execute each listed node in order"; nothing observes or enforces the walk. There are no cycles, no mid-path branching, no typed edges, no pre/post-conditions, no artifact or data flow between nodes. So claims such as "context-dependent transitions" and "different task conditions induce different execution paths" describe intended behavior that the artifact does not enforce — a two-level SOP library plus a router would be an accurate description. Either rename and drop the graph-theoretic framing (and the corresponding claims about a "structured search space"), or actually implement traversal — e.g. force the agent to commit to a workflow and emit node-by-node steps with the harness checking compliance — which would be a far more distinctive contribution and would also give you a direct measurement of whether structure is followed.

W2. The operators are not structure-aware, so the "structure-aware mutation/crossover" contribution is not operationalized

All four operators are whole-document regeneration prompts that differ only in which section they may edit; the "implementation will enforce this boundary" is a post-hoc section parse. "Crossover" is an LLM merging prose, not recombination over graph structure: no node-level crossover that respects dependencies, no typed mutation restricted to admissible node/edge edits, no graph edit distance, no topology constraints beyond name-consistency checked by a format validator (Appendix B.3 checks that referenced nodes exist and declared nodes are used). As written, the search is section-scoped LLM rewriting, and the reported gains cannot be attributed to structure-aware search. Please show per-generation diffs (nodes added / removed / rewired, edges changed) and implement at least one operator that acts on graph elements rather than on the document.

W3. The ablations are confounded, and one is close to a strawman

  • w/o mutation (Table 3, −17.02 average) leaves only crossover. Crossover cannot introduce new content, so after initialization the population can only recombine what already exists: performance collapse is a foregone conclusion, not evidence that "execution feedback matters". The informative control is replacing mutation with a non-evolutionary reflection step (SkillOpt-style patch) at matched budget.
  • Operator budget is entangled with operator type. With N = 4 and round-robin scheduling, every generation is exactly {1 global-mut, 1 graph-mut, 1 global-cross, 1 graph-cross}. Removing an operator also changes the per-generation operator mix (e.g. w/o crossover yields 2 mutations/generation), so Table 3 confounds operator presence with variation budget and with the mutation:crossover ratio.
  • w/o population (N = 1) also removes crossover by construction, so "population selection matters" and "crossover matters" are not separated. A 2×2 {N = 1, N = 4} × {with, without crossover} at a matched number of generated candidates would fix this.

W4. Generality and adoption: the paper never tests the path it claims to matter for

This is, for me, the decisive practical objection.

  • Bespoke schema. The method requires skills to be authored in a specific three-section markdown schema, and the initializer, all four operators, the validator and the "execution guide" text all presume it. In practice skills are packaged as SKILL.md plus optional resources (scripts, templates, examples) and shared across harnesses — see the SkillsBench specification (§2.1), where portability and cross-harness reuse are defining properties of a skill. The paper offers no experiment on the adoption path: nothing converts an existing human-authored skill into the graph form and reports conversion cost, information loss, or validator/regeneration failure rate; nothing tests whether users actually maintain graph skills more easily than text skills (a stated selling point); every artifact evaluated was generated by the method itself. At minimum: report the graph-validation failure/regeneration rate, and convert a public curated skill corpus (e.g. the SkillsBench skills) to your format and report graph-vs-SKILL.md deltas.
  • Format sensitivity cuts against the premise, not for it. The paper's motivation is that text skills are fragile and may not transfer. But graph formatting is itself a format choice, and prompt-format sensitivity is known to be large and model-specific (Sclar et al., ICLR 2024). A bespoke serialization therefore inherits the same risk it claims to solve; portability must be demonstrated, not assumed from the fact that the medium is natural language. Dynamic Cheatsheet (Suzgun et al., 2025) is directly relevant: accumulated textual memory that helps one model can hurt another, so "it transfers" needs a model matrix, not a pair.
  • The transfer study tests only the easy direction. Table 4 optimizes on GPT-5.4-nano and deploys on GPT-5.4 (weak→strong), within one family, same harness, same benchmark, 3 benchmarks. The operationally relevant direction is the reverse (strong→cheap deployment), and cross-family transfer is absent entirely. Note also that the transfer deltas on SearchQA (+0.27 over Direct) and DocVQA (−0.54 over Direct) are essentially "no worse", and the single clear win (Spreadsheet transferred 71.78 > direct 69.40) is unexplained — with |D_val| = 40, a skill optimized on the weaker model beating the one optimized on the stronger model is at least as likely to be selection noise as a finding.
  • The artifacts are benchmark-specific. The optimized spreadsheet skill contains instructions such as resolving "the exact answer cell … named by the … benchmark answer position" and reasoning about "SpreadsheetBench evaluation". These are evaluation-protocol facts, not transferable spreadsheet procedure, and they are exactly the kind of content that will not generalize to a real user's workbook. This also undercuts the "reusable procedural knowledge" framing in the introduction.

W5. The evaluation cannot support the paper's own small-model motivation

The stated motivation is that unstructured skills are hard for small models to follow and that graph structure fixes this. The evidence is one within-family 2×2 (same provider, unknown parameter counts and architectures) plus a 2.25-point difference in average deltas over 3 seeds. Missing, in order of importance:

  1. A capability ladder. nano/mini/full within one family, or open-weight models spanning ~1B–32B, with Δ reported as a function of capability. Two cost tiers of one model family is not a model-size study.
  2. No architecture or family variation. Dense vs. MoE, reasoning vs. non-reasoning, and at least two non-OpenAI families (the skill-utilization behavior of a harness/model pair is not portable — SkillsBench finds Codex CLI frequently ignores provided skills, which directly threatens the Codex row here).
  3. Reasoning effort is pinned to "medium" and never varied. Model name is the only capability knob; test-time compute is held fixed, even though reasoning budget mediates precisely the behaviors the paper claims to improve — parsing a long skill document, planning a step sequence, and recovering from a failed step. Because both capability and cost in a model family are largely set by the reasoning budget, fixing it at medium makes "capability" a confounded composite (nano at medium effort can plausibly match full at low effort), so "less capable models benefit most" may be a statement about reasoning budget rather than about model size. It is also the cheapest experiment in this review to run: a 2 models × {low, medium, high} grid on one or two benchmarks would show whether the graph advantage is (a) an instruction-parsing effect that vanishes once the model plans well on its own, or (b) a scaffolding effect that grows as the budget shrinks. Either outcome is informative; a single setting cannot distinguish them, and a skill whose benefit exists only at one effort level is not portable in the sense the paper claims.
  4. No interaction test and no CIs. The claim is an interaction claim (method × capability) supported by comparing two averaged deltas, with no test for the interaction and no interval on either average; for calibration, the run-to-run standard deviations reported in Table 6 for the same benchmarks reach 6.4 points.
  5. The mechanism is never measured. Nothing measures step adherence, rule recall, or sensitivity to skill length, so "less capable LLMs struggle with overlong instructions" is asserted from long-context intuition (cf. Liu et al., TACL 2024) rather than tested. Two cheap probes: (a) report skill token counts and a length-matched control; (b) measure the fraction of workflow steps the agent actually executes, by model.
  6. An unexcluded generic explanation. Weaker models typically benefit more from any scaffolding. The differential could be a capability effect rather than a graph effect. The test is straightforward: run a second, non-graph text optimizer (GEPA, ProTeGi) on both models; if it also shows a larger nano gain, the graph is not the cause.
  7. The pattern is fragile. The differential flips sign on LiveMath (nano −0.80 vs. GPT-5.4 +1.07) — precisely where |D_val| = 18 and the honest interpretation is selection noise (see W7). Meanwhile Table 1's own headline gains are described as "small and less capable models benefit the most" without reporting normalized gain, which matters because absolute point deltas are mechanically larger for the model with more headroom.
  8. The two "capability tiers" are not a clean capability manipulation. The no-skill gap between the two models is 4.0 points on SpreadsheetBench (39.16 vs. 35.12) but 42.3 points on DocVQA (79.05 vs. 36.72) and 19.4 on SearchQA. A pair of models that differs by 4 points on one benchmark and 42 on another is not differing (only) in general capability, so no ordering of "weaker/stronger" derived from this pair is well defined — and yet the small-model conclusion is drawn by averaging deltas across exactly these heterogeneous gaps. Worth reporting the per-benchmark capability gap alongside every delta, and explaining the DocVQA gap (a 42-point gap and a +44.2-point skill gain on a document-VQA task look more like a broken baseline/answer-format issue in the nano configuration than like procedural knowledge transfer).

W6. The task suite never leaves short-horizon, single-artifact settings, so the mechanism is never stressed

Every benchmark is a one-shot, single-artifact task: answer a question given its evidence (SearchQA), edit one workbook through generated Python (SpreadsheetBench), answer one question about one document image (DocVQA), pick A–D on one problem (LiveMath), and finish one household episode within 50 steps (ALFWorld, six task types). Three of the five are single-turn in the no-harness row, i.e. no tool use at all. None of the five requires multi-file or repository-level change against a test suite (SWE-bench, Terminal-Bench), multi-turn tool–agent–user interaction under policy constraints (τ-bench), desktop/OS state manipulation (OSWorld), or long-horizon planning with tool discovery and partial-failure recovery (GAIA). The Codex row comes closest to an agentic setting, but it covers only the four non-ALFWorld benchmarks and allows a single code-generation round in a single local workspace — a minimal agent loop, not agentic coding — while ALFWorld, the only sequential environment, appears only in the no-harness row.

This is not merely a coverage complaint; it is a mismatch with the paper's own thesis. The benefit claimed for explicit workflow guidance is that the agent knows which step it is in, does not drift, and does not compound errors — phenomena that require horizon to appear. A 4–6 step ALFWorld episode or a single-file script edit leaves almost no room to drift, so the design is evaluated precisely where it has least to offer. Conversely, if graph-structured workflow guidance does help, the effect should be largest on long-horizon, many-tool tasks where the agent must repeatedly re-decide its position in a procedure; that is the natural confirmatory experiment, and it is absent.

Two consequences worth stating separately:

  • Routing is never tested at scale. The "Use when" mechanism is supposed to select among workflows, but the largest skill shown in the appendix has 16 nodes and 5 workflows, shaped by a taxonomy the authors themselves defined for one benchmark, and every benchmark uses exactly one skill. Nothing tests the regime where a graph representation would actually earn its keep — routing over a library of dozens of skills and hundreds of nodes, where a wrong branch is both likely and costly.
  • Difficulty range is compressed. With no-skill accuracy already at 84% (SearchQA) and 90% (DocVQA), the measurable headroom is small and the reported deltas are squeezed against a ceiling; hard tasks typically have low baselines, which is where methods separate and where a 1–2 point aggregate difference would be interpretable rather than within noise.

W7. Statistical and cost hygiene

  • Table 1 vs. Table 6 disagree for identical settings. ALFWorld / SkillOpt: 57.46 in Table 1 vs. 52.46 (mean of 10) in Table 6. DocVQA / GraphSkillEvo: 80.92 vs. 78.72. SearchQA / SkillOpt: 69.52 vs. 69.44. Please reconcile, and state whether Table 1's 3 runs are a subset of Table 6's 10.
  • No uncertainty in the main table, 14 uncorrected comparisons, significance tested on one model and 4 benchmarks, with DocVQA at p = 0.053.
  • Budget parity is not established. Tokens are reported, but the dominant cost in this pipeline is fitness evaluation (≈24 candidate skills × |D_val| per run) plus rollouts, and these are not reported. Report #LLM calls, #rollouts, #validation evaluations, wall-clock and USD, and provide a score-vs-rollouts Pareto curve; Figure 3 is validation score vs. tokens on one model only. Also state whether hidden reasoning tokens are included in the 61.94M / 81.08M totals: GraphSkillEvo issues far more model calls per run than SkillOpt (four candidate skills per generation, per-parent rollouts, and full-validation scoring of every candidate), so a token comparison that counts reasoning tokens inconsistently across the two methods is not interpretable.
  • Validation-set overfitting is unaddressed and is first-order here. The returned skill is the argmax over a final population of 4 that has survived five rounds of truncation selection on the same validation set (20 generated candidates per run), evaluated on |D_val| ∈ {18, 18, 40, 53}. At |D_val| = 18 and p ≈ 0.25 the standard error of a validation score is ≈10 points, so each round of "top-4 of 8" is substantially a coin flip, and the survivors accumulate selection bias. This affects LiveMath and ALFWorld most — the two benchmarks where the conclusions are weakest. Report (i) the population-mean test score, (ii) the gap between the final-generation best and the second best, and (iii) an estimate from a nested/held-out selection split.
  • Test-set noise vs. claim size. LiveMath's 124 items and ALFWorld's 134 give binomial standard errors of ≈4 points. The ALFWorld +3.73 (nano) and the 2.25-point model differential are of the same order as that noise, so neither difference is individually decisive.

W8. Benchmarks, baselines and analysis

  • No transfer to a related but unseen task, and no use of the benchmark built for exactly this question. There is one dataset per task family, so nothing tests whether an optimized skill helps on tasks it was not optimized on, which is the property that makes a skill reusable. SkillsBench (86 tasks, 11 domains, deterministic verifiers) was built to answer precisely this and is cited by the paper only for its definition of a skill; its central finding directly challenges this pipeline: self-generated skills provide no benefit on average, and −5.6 pp for Codex + GPT-5.2. A method whose every artifact is LLM-generated should engage with that result — e.g. by reporting curated-skill vs. optimized-skill deltas on the same tasks.
  • ALFWorld seen/unseen. Please state whether optimization and test draw on the same six task types. If so, the "Use when" routing is effectively fitted to the benchmark's task taxonomy, and the reported ALFWorld gains partly measure taxonomy memorization.
  • Protocol leakage. On SpreadsheetBench both the human-written and the one-shot LLM skill reduce accuracy relative to no skill (−1.31 and −3.57 for GPT-5.4), while skill optimization yields +25.7 (SkillOpt) and +30.2 (GraphSkillEvo). Combined with the protocol-specific node text noted in W4, this suggests a substantial share of the gain is recovery of interface/evaluation details rather than procedural expertise. An ablation that strips protocol-specific advice (and a harness whose native prompt already contains those details) would separate the two.
  • The "procedural tasks benefit most" reading is a post-hoc partition of five benchmarks and is contradicted inside the same table. On GPT-5.4 the largest non-spreadsheet gain is LiveMath (+15.05), a multiple-choice math task, which exceeds ALFWorld (+13.93); DocVQA (+11.32), a factual task, is within 3 points of ALFWorld. With one dataset per family and post-hoc categorization, the claim that gains concentrate in procedural benchmarks is not identifiable.
  • Missing baselines. The contribution is a population-based evolutionary text optimizer, yet no EA-for-prompts baseline is cited or run: EvoPrompt, Promptbreeder, Language Model Crossover (the direct precedent for LLM-implemented crossover), GEPA, QDAIF, ProTeGi, OPRO, TextGrad, DSPy/MIPROv2. Also absent: the skill-optimization methods the paper itself cites (EvoSkill, Trace2Skill, SkillX, AutoSkill, CoevoSKills), Agent Workflow Memory (workflow-level procedural knowledge induced from trajectories — the closest prior on this axis), an equal-budget reflexion/best-of-N control, a length-matched placebo skill, and a genuinely expert-written skill for at least one benchmark (the current "human skill" is worse than no skill on SpreadsheetBench for both models, and on LiveMath for GPT-5.4, so it cannot serve as a reference point).
  • The two harness rows use different protocols, so they are not comparable: 30 code-generation rounds per task without a harness versus a single round with Codex (Appendix B.1). A skill can therefore substitute for repair rounds in one row and not the other, and a large part of the cross-harness difference in skill benefit may be protocol, not harness. Please report an equal-round control (e.g. 1 round in both, or 30 in both) and an explicit measure of whether the Codex agent reads and follows the skill at all.
  • Missing analyses. Skill length, node/edge counts and change rates across generations — needed even to define "low redundancy" and "compact search space", neither of which is measured; a redundancy metric (e.g. n-gram overlap across nodes); routing accuracy (does the agent select the intended workflow?); an error taxonomy across all five benchmarks rather than one qualitative ALFWorld case study of successes; the marginal value of each operator at matched candidate count; and the inference-time token/latency overhead of the skill. The "compact search space" claim is also unargued: the operators regenerate whole documents, so the optimizer still searches over token sequences — the structure constrains output form, not the size of the search space.
Questions for authors
  1. What is structure-aware about your operators, concretely? Please show a parent→child diff and the per-generation counts of added/removed/rewired nodes and edges.
  2. Can you provide the information- and length-matched flat-SOP control (with "Use when" routing retained), and a step-permuted variant? Does the gap survive?
  3. How large is the selection bias from five rounds of "top-4 of 8" on an 18–53-example validation set? Please report the population-mean test score and a nested-selection estimate.
  4. What are the #rollouts, #validation evaluations, #LLM calls and USD cost per setting; is reasoning-token consumption included in the reported totals; and what does the score-vs-rollouts curve look like?
  5. Can you report a model ladder (≥3 sizes × ≥2 families, including open weights) and, on the same protocol, a reasoning-effort sweep ({low, medium, high} × 3 models), each with an interaction test for capability × method?
  6. Why is reasoning effort fixed at "medium"? Which reading of the mechanism do you expect — the graph helping most when the model reasons less (scaffolding) or least (redundant)? A 2×3 grid on two benchmarks would settle it.
  7. Can you add at least one long-horizon agentic-coding benchmark (SWE-bench Verified or Terminal-Bench) under the same optimization protocol? If explicit workflow guidance is the mechanism, the benefit should be largest where agents drift — and a single-code-round Codex row does not exercise that regime.
  8. The "Use when" router is never tested beyond five hand-specified workflows or beyond one skill per benchmark. What happens with a library of 20–50 skills, or on a task whose correct workflow is absent from the initialized taxonomy?
  9. Was ALFWorld evaluated on seen or unseen task types, and how are the six task types distributed across train/val/test?
  10. Please explain the Table 1 / Table 6 discrepancies (ALFWorld SkillOpt 57.46 vs. 52.46; DocVQA GraphSkillEvo 80.92 vs. 78.72).
  11. What happens when you convert an existing curated skill corpus to your format — gain or loss relative to the original SKILL.md, and at what authoring/validation cost?
  12. Does the Codex agent actually read the skill file? Please report a placebo-skill control and a skill-utilization measure, given that Codex CLI is known to neglect provided skills.
  13. Does the SpreadsheetBench gain survive removal of protocol-specific guidance ("benchmark answer position", "SpreadsheetBench evaluation", the static-value-vs-formula advice)?
  14. Why would a skill optimized on GPT-5.4-nano transfer better to GPT-5.4 than one optimized on GPT-5.4?
  15. How often does graph-structure validation fail and require regeneration, and what fraction of optimization tokens does that consume?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.