W1. The central claim — graph vs. unstructured at matched information — is never tested, and the authors say so
Section 5.1 builds the "unstructured counterpart" by removing explicit workflow organization "while retaining task-general guidance and node-level instructions", and Appendix E.4 concedes that this "examines the contribution of that explicit workflow guidance, rather than comparing graph and non-graph representations that preserve equivalent workflow information." That is exactly the comparison the paper's contribution depends on. The unstructured control therefore has strictly less information: it loses the ordering and the "use when" routing, i.e. it is precisely the "lengthy unordered bullet list" the paper criticizes in the introduction. Table 2 (−4.52/−2.50/−4.19/−1.35/−0.75 on SearchQA/Spreadsheet/DocVQA/LiveMath/ALFWorld) and the w/o graph structure row of Table 3 (−4.10/−10.12/−8.11) consequently cannot distinguish
- "graph organization helps" from
- "an ordered procedure with task-type routing helps" (which is not a graph claim at all), from
- "longer, more specific guidance helps".
Required controls. (i) A content-matched, length-matched flattening: identical node text, identical "Use when" lines, rendered as Step 1 … Step N under each task type, at the same token budget. (ii) A step-permuted version of that flat SOP, to isolate the contribution of ordering specifically. (iii) A length-matched placebo skill (generic guidance of the same size), to bound the "any extra guidance helps" effect. Without (i)–(iii) the headline representation claim is unsupported.
A related conceptual issue. The "graph" is a document layout, not an executable structure. The agent is asked in prose to "select the graph using its Use when line, then execute each listed node in order"; nothing observes or enforces the walk. There are no cycles, no mid-path branching, no typed edges, no pre/post-conditions, no artifact or data flow between nodes. So claims such as "context-dependent transitions" and "different task conditions induce different execution paths" describe intended behavior that the artifact does not enforce — a two-level SOP library plus a router would be an accurate description. Either rename and drop the graph-theoretic framing (and the corresponding claims about a "structured search space"), or actually implement traversal — e.g. force the agent to commit to a workflow and emit node-by-node steps with the harness checking compliance — which would be a far more distinctive contribution and would also give you a direct measurement of whether structure is followed.
W2. The operators are not structure-aware, so the "structure-aware mutation/crossover" contribution is not operationalized
All four operators are whole-document regeneration prompts that differ only in which section they may edit; the "implementation will enforce this boundary" is a post-hoc section parse. "Crossover" is an LLM merging prose, not recombination over graph structure: no node-level crossover that respects dependencies, no typed mutation restricted to admissible node/edge edits, no graph edit distance, no topology constraints beyond name-consistency checked by a format validator (Appendix B.3 checks that referenced nodes exist and declared nodes are used). As written, the search is section-scoped LLM rewriting, and the reported gains cannot be attributed to structure-aware search. Please show per-generation diffs (nodes added / removed / rewired, edges changed) and implement at least one operator that acts on graph elements rather than on the document.
W3. The ablations are confounded, and one is close to a strawman
w/o mutation (Table 3, −17.02 average) leaves only crossover. Crossover cannot introduce new content, so after initialization the population can only recombine what already exists: performance collapse is a foregone conclusion, not evidence that "execution feedback matters". The informative control is replacing mutation with a non-evolutionary reflection step (SkillOpt-style patch) at matched budget.
- Operator budget is entangled with operator type. With N = 4 and round-robin scheduling, every generation is exactly {1 global-mut, 1 graph-mut, 1 global-cross, 1 graph-cross}. Removing an operator also changes the per-generation operator mix (e.g.
w/o crossover yields 2 mutations/generation), so Table 3 confounds operator presence with variation budget and with the mutation:crossover ratio.
w/o population (N = 1) also removes crossover by construction, so "population selection matters" and "crossover matters" are not separated. A 2×2 {N = 1, N = 4} × {with, without crossover} at a matched number of generated candidates would fix this.
W4. Generality and adoption: the paper never tests the path it claims to matter for
This is, for me, the decisive practical objection.
- Bespoke schema. The method requires skills to be authored in a specific three-section markdown schema, and the initializer, all four operators, the validator and the "execution guide" text all presume it. In practice skills are packaged as
SKILL.md plus optional resources (scripts, templates, examples) and shared across harnesses — see the SkillsBench specification (§2.1), where portability and cross-harness reuse are defining properties of a skill. The paper offers no experiment on the adoption path: nothing converts an existing human-authored skill into the graph form and reports conversion cost, information loss, or validator/regeneration failure rate; nothing tests whether users actually maintain graph skills more easily than text skills (a stated selling point); every artifact evaluated was generated by the method itself. At minimum: report the graph-validation failure/regeneration rate, and convert a public curated skill corpus (e.g. the SkillsBench skills) to your format and report graph-vs-SKILL.md deltas.
- Format sensitivity cuts against the premise, not for it. The paper's motivation is that text skills are fragile and may not transfer. But graph formatting is itself a format choice, and prompt-format sensitivity is known to be large and model-specific (Sclar et al., ICLR 2024). A bespoke serialization therefore inherits the same risk it claims to solve; portability must be demonstrated, not assumed from the fact that the medium is natural language. Dynamic Cheatsheet (Suzgun et al., 2025) is directly relevant: accumulated textual memory that helps one model can hurt another, so "it transfers" needs a model matrix, not a pair.
- The transfer study tests only the easy direction. Table 4 optimizes on GPT-5.4-nano and deploys on GPT-5.4 (weak→strong), within one family, same harness, same benchmark, 3 benchmarks. The operationally relevant direction is the reverse (strong→cheap deployment), and cross-family transfer is absent entirely. Note also that the transfer deltas on SearchQA (+0.27 over Direct) and DocVQA (−0.54 over Direct) are essentially "no worse", and the single clear win (Spreadsheet transferred 71.78 > direct 69.40) is unexplained — with |D_val| = 40, a skill optimized on the weaker model beating the one optimized on the stronger model is at least as likely to be selection noise as a finding.
- The artifacts are benchmark-specific. The optimized spreadsheet skill contains instructions such as resolving "the exact answer cell … named by the … benchmark answer position" and reasoning about "SpreadsheetBench evaluation". These are evaluation-protocol facts, not transferable spreadsheet procedure, and they are exactly the kind of content that will not generalize to a real user's workbook. This also undercuts the "reusable procedural knowledge" framing in the introduction.
W5. The evaluation cannot support the paper's own small-model motivation
The stated motivation is that unstructured skills are hard for small models to follow and that graph structure fixes this. The evidence is one within-family 2×2 (same provider, unknown parameter counts and architectures) plus a 2.25-point difference in average deltas over 3 seeds. Missing, in order of importance:
- A capability ladder. nano/mini/full within one family, or open-weight models spanning ~1B–32B, with Δ reported as a function of capability. Two cost tiers of one model family is not a model-size study.
- No architecture or family variation. Dense vs. MoE, reasoning vs. non-reasoning, and at least two non-OpenAI families (the skill-utilization behavior of a harness/model pair is not portable — SkillsBench finds Codex CLI frequently ignores provided skills, which directly threatens the Codex row here).
- Reasoning effort is pinned to "medium" and never varied. Model name is the only capability knob; test-time compute is held fixed, even though reasoning budget mediates precisely the behaviors the paper claims to improve — parsing a long skill document, planning a step sequence, and recovering from a failed step. Because both capability and cost in a model family are largely set by the reasoning budget, fixing it at medium makes "capability" a confounded composite (nano at medium effort can plausibly match full at low effort), so "less capable models benefit most" may be a statement about reasoning budget rather than about model size. It is also the cheapest experiment in this review to run: a 2 models × {low, medium, high} grid on one or two benchmarks would show whether the graph advantage is (a) an instruction-parsing effect that vanishes once the model plans well on its own, or (b) a scaffolding effect that grows as the budget shrinks. Either outcome is informative; a single setting cannot distinguish them, and a skill whose benefit exists only at one effort level is not portable in the sense the paper claims.
- No interaction test and no CIs. The claim is an interaction claim (method × capability) supported by comparing two averaged deltas, with no test for the interaction and no interval on either average; for calibration, the run-to-run standard deviations reported in Table 6 for the same benchmarks reach 6.4 points.
- The mechanism is never measured. Nothing measures step adherence, rule recall, or sensitivity to skill length, so "less capable LLMs struggle with overlong instructions" is asserted from long-context intuition (cf. Liu et al., TACL 2024) rather than tested. Two cheap probes: (a) report skill token counts and a length-matched control; (b) measure the fraction of workflow steps the agent actually executes, by model.
- An unexcluded generic explanation. Weaker models typically benefit more from any scaffolding. The differential could be a capability effect rather than a graph effect. The test is straightforward: run a second, non-graph text optimizer (GEPA, ProTeGi) on both models; if it also shows a larger nano gain, the graph is not the cause.
- The pattern is fragile. The differential flips sign on LiveMath (nano −0.80 vs. GPT-5.4 +1.07) — precisely where |D_val| = 18 and the honest interpretation is selection noise (see W7). Meanwhile Table 1's own headline gains are described as "small and less capable models benefit the most" without reporting normalized gain, which matters because absolute point deltas are mechanically larger for the model with more headroom.
- The two "capability tiers" are not a clean capability manipulation. The no-skill gap between the two models is 4.0 points on SpreadsheetBench (39.16 vs. 35.12) but 42.3 points on DocVQA (79.05 vs. 36.72) and 19.4 on SearchQA. A pair of models that differs by 4 points on one benchmark and 42 on another is not differing (only) in general capability, so no ordering of "weaker/stronger" derived from this pair is well defined — and yet the small-model conclusion is drawn by averaging deltas across exactly these heterogeneous gaps. Worth reporting the per-benchmark capability gap alongside every delta, and explaining the DocVQA gap (a 42-point gap and a +44.2-point skill gain on a document-VQA task look more like a broken baseline/answer-format issue in the nano configuration than like procedural knowledge transfer).
W6. The task suite never leaves short-horizon, single-artifact settings, so the mechanism is never stressed
Every benchmark is a one-shot, single-artifact task: answer a question given its evidence (SearchQA), edit one workbook through generated Python (SpreadsheetBench), answer one question about one document image (DocVQA), pick A–D on one problem (LiveMath), and finish one household episode within 50 steps (ALFWorld, six task types). Three of the five are single-turn in the no-harness row, i.e. no tool use at all. None of the five requires multi-file or repository-level change against a test suite (SWE-bench, Terminal-Bench), multi-turn tool–agent–user interaction under policy constraints (τ-bench), desktop/OS state manipulation (OSWorld), or long-horizon planning with tool discovery and partial-failure recovery (GAIA). The Codex row comes closest to an agentic setting, but it covers only the four non-ALFWorld benchmarks and allows a single code-generation round in a single local workspace — a minimal agent loop, not agentic coding — while ALFWorld, the only sequential environment, appears only in the no-harness row.
This is not merely a coverage complaint; it is a mismatch with the paper's own thesis. The benefit claimed for explicit workflow guidance is that the agent knows which step it is in, does not drift, and does not compound errors — phenomena that require horizon to appear. A 4–6 step ALFWorld episode or a single-file script edit leaves almost no room to drift, so the design is evaluated precisely where it has least to offer. Conversely, if graph-structured workflow guidance does help, the effect should be largest on long-horizon, many-tool tasks where the agent must repeatedly re-decide its position in a procedure; that is the natural confirmatory experiment, and it is absent.
Two consequences worth stating separately:
- Routing is never tested at scale. The "Use when" mechanism is supposed to select among workflows, but the largest skill shown in the appendix has 16 nodes and 5 workflows, shaped by a taxonomy the authors themselves defined for one benchmark, and every benchmark uses exactly one skill. Nothing tests the regime where a graph representation would actually earn its keep — routing over a library of dozens of skills and hundreds of nodes, where a wrong branch is both likely and costly.
- Difficulty range is compressed. With no-skill accuracy already at 84% (SearchQA) and 90% (DocVQA), the measurable headroom is small and the reported deltas are squeezed against a ceiling; hard tasks typically have low baselines, which is where methods separate and where a 1–2 point aggregate difference would be interpretable rather than within noise.
W7. Statistical and cost hygiene
- Table 1 vs. Table 6 disagree for identical settings. ALFWorld / SkillOpt: 57.46 in Table 1 vs. 52.46 (mean of 10) in Table 6. DocVQA / GraphSkillEvo: 80.92 vs. 78.72. SearchQA / SkillOpt: 69.52 vs. 69.44. Please reconcile, and state whether Table 1's 3 runs are a subset of Table 6's 10.
- No uncertainty in the main table, 14 uncorrected comparisons, significance tested on one model and 4 benchmarks, with DocVQA at p = 0.053.
- Budget parity is not established. Tokens are reported, but the dominant cost in this pipeline is fitness evaluation (≈24 candidate skills × |D_val| per run) plus rollouts, and these are not reported. Report #LLM calls, #rollouts, #validation evaluations, wall-clock and USD, and provide a score-vs-rollouts Pareto curve; Figure 3 is validation score vs. tokens on one model only. Also state whether hidden reasoning tokens are included in the 61.94M / 81.08M totals: GraphSkillEvo issues far more model calls per run than SkillOpt (four candidate skills per generation, per-parent rollouts, and full-validation scoring of every candidate), so a token comparison that counts reasoning tokens inconsistently across the two methods is not interpretable.
- Validation-set overfitting is unaddressed and is first-order here. The returned skill is the argmax over a final population of 4 that has survived five rounds of truncation selection on the same validation set (20 generated candidates per run), evaluated on |D_val| ∈ {18, 18, 40, 53}. At |D_val| = 18 and p ≈ 0.25 the standard error of a validation score is ≈10 points, so each round of "top-4 of 8" is substantially a coin flip, and the survivors accumulate selection bias. This affects LiveMath and ALFWorld most — the two benchmarks where the conclusions are weakest. Report (i) the population-mean test score, (ii) the gap between the final-generation best and the second best, and (iii) an estimate from a nested/held-out selection split.
- Test-set noise vs. claim size. LiveMath's 124 items and ALFWorld's 134 give binomial standard errors of ≈4 points. The ALFWorld +3.73 (nano) and the 2.25-point model differential are of the same order as that noise, so neither difference is individually decisive.
W8. Benchmarks, baselines and analysis
- No transfer to a related but unseen task, and no use of the benchmark built for exactly this question. There is one dataset per task family, so nothing tests whether an optimized skill helps on tasks it was not optimized on, which is the property that makes a skill reusable. SkillsBench (86 tasks, 11 domains, deterministic verifiers) was built to answer precisely this and is cited by the paper only for its definition of a skill; its central finding directly challenges this pipeline: self-generated skills provide no benefit on average, and −5.6 pp for Codex + GPT-5.2. A method whose every artifact is LLM-generated should engage with that result — e.g. by reporting curated-skill vs. optimized-skill deltas on the same tasks.
- ALFWorld seen/unseen. Please state whether optimization and test draw on the same six task types. If so, the "Use when" routing is effectively fitted to the benchmark's task taxonomy, and the reported ALFWorld gains partly measure taxonomy memorization.
- Protocol leakage. On SpreadsheetBench both the human-written and the one-shot LLM skill reduce accuracy relative to no skill (−1.31 and −3.57 for GPT-5.4), while skill optimization yields +25.7 (SkillOpt) and +30.2 (GraphSkillEvo). Combined with the protocol-specific node text noted in W4, this suggests a substantial share of the gain is recovery of interface/evaluation details rather than procedural expertise. An ablation that strips protocol-specific advice (and a harness whose native prompt already contains those details) would separate the two.
- The "procedural tasks benefit most" reading is a post-hoc partition of five benchmarks and is contradicted inside the same table. On GPT-5.4 the largest non-spreadsheet gain is LiveMath (+15.05), a multiple-choice math task, which exceeds ALFWorld (+13.93); DocVQA (+11.32), a factual task, is within 3 points of ALFWorld. With one dataset per family and post-hoc categorization, the claim that gains concentrate in procedural benchmarks is not identifiable.
- Missing baselines. The contribution is a population-based evolutionary text optimizer, yet no EA-for-prompts baseline is cited or run: EvoPrompt, Promptbreeder, Language Model Crossover (the direct precedent for LLM-implemented crossover), GEPA, QDAIF, ProTeGi, OPRO, TextGrad, DSPy/MIPROv2. Also absent: the skill-optimization methods the paper itself cites (EvoSkill, Trace2Skill, SkillX, AutoSkill, CoevoSKills), Agent Workflow Memory (workflow-level procedural knowledge induced from trajectories — the closest prior on this axis), an equal-budget reflexion/best-of-N control, a length-matched placebo skill, and a genuinely expert-written skill for at least one benchmark (the current "human skill" is worse than no skill on SpreadsheetBench for both models, and on LiveMath for GPT-5.4, so it cannot serve as a reference point).
- The two harness rows use different protocols, so they are not comparable: 30 code-generation rounds per task without a harness versus a single round with Codex (Appendix B.1). A skill can therefore substitute for repair rounds in one row and not the other, and a large part of the cross-harness difference in skill benefit may be protocol, not harness. Please report an equal-round control (e.g. 1 round in both, or 30 in both) and an explicit measure of whether the Codex agent reads and follows the skill at all.
- Missing analyses. Skill length, node/edge counts and change rates across generations — needed even to define "low redundancy" and "compact search space", neither of which is measured; a redundancy metric (e.g. n-gram overlap across nodes); routing accuracy (does the agent select the intended workflow?); an error taxonomy across all five benchmarks rather than one qualitative ALFWorld case study of successes; the marginal value of each operator at matched candidate count; and the inference-time token/latency overhead of the skill. The "compact search space" claim is also unargued: the operators regenerate whole documents, so the optimizer still searches over token sequences — the structure constrains output form, not the size of the search space.