- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
SkillWorld has an LLM extract a typed manifest for each skill (capability, ports, preconditions, effects, constraints) against a shared ontology. Fixed Datalog rules (Soufflé) then compile five provenance-carrying relation types: CanFeed, Enables, AlternativeTo, Specializes and ConflictsWith. At runtime, a deterministic three-valued (Strong–Kleene) projection labels each relation confirmed, unknown or rejected under the current symbolic state and resource bindings. An LLM then selects skills from a relation-annotated candidate pool expanded from BM25 seeds. Between releases, "governed self-evolution" proposes ontology edits from manifest evidence and compilation diagnostics. An LLM picks an edit or abstains, a Lean-verified checker confirms source consistency, affected manifests are re-extracted, and a checked successor release is published atomically. Lean 4 proofs cover projection soundness and the authorization and publication invariants. Results: on 72 SkillsBench tasks, the evolved graph reaches 54.2% (Qwen) and 66.7% (Opus), 15.3/11.1 points above the best baseline. On ALFWorld it reaches 85.1–85.8% SR@15 with 43–72% fewer invalid actions than BM25. On 36 internal-service tasks it reaches 72.2%.
Critical strengths
- Separating a release-level "potential" relation graph from its state-conditioned applicability, with explicit unknowns under partial observation and rule-based derivation with provenance, is a clean and reusable idea for skill graphs. The ALFWorld ablations (no relation labels, static or random state) support it, with consistent reductions in invalid actions.
- The system is engineered with unusual rigor: mechanized Lean proofs, recompilation that propagates ontology corrections, and atomic publication. It is evaluated in three settings, including a real deployment.
Critical weaknesses
- Evolution drives all of the SkillsBench gain, but the paper does not explain why (beyond one traced example in App. G) or rule out leakage. The base graph is on par with the baselines (36.1 vs 33.3–38.9 for Qwen; 54.2 vs 50.0–55.6 for Opus). The 12–18 point improvement comes entirely from 41–51 evolution rounds (168–379 edits). The paper does not show which edits matter or how retrieved top-5 sets change on newly solved tasks. It also does not state clearly whether any SkillsBench task, instruction, verifier or outcome informs evolution: Appendix D mentions checking "configured task cases" in CHECKEDINPUT. Appendix G indicates that the Qwen run's evolution calls use qwen3.8-max, i.e., each evaluated model evolves its own graph, so it is unclear whether the evolved graph is generally better or tuned to that model. Governance is the claimed novelty, yet there is no ablation against ungoverned LLM ontology edits, and the token/time cost of evolution is not reported (only the budget mechanism is described). The Lean results certify the three-valued projection logic and the authorization/publication bookkeeping (tokens, CAS, frame conditions), but not the semantic accuracy of the LLM-extracted manifests and the relations derived from them, which Appendix A treats as an assumption ("the accuracy of manifest extraction and grounding determines the reliability of decided judgments"). So the formal machinery's contribution to task success is not established.
- The evaluation design leaves comparisons confounded or under-powered. SkillWorld uses an LLM selector over structured relation context, while several baselines (BM25, vector, co-use graph) rank by retrieval score alone. BM25 and BM25+LLM reranking give identical results for both models (25/72, 37/72), which should be checked. There is no CI on 72 tasks × 3 runs, where gaps among baselines are 1–4 tasks. On ALFWorld several conditions are identical across all three repeats (e.g., Qwen Full 114/114/114, BM25 101/101/101), so the repeats are not independent samples; per-game paired tests are needed for the 1.5–4 pp gaps to SkillDAG. The internal-service tasks are drawn from SkillWorld's own deployment traces, which favors the deployed system, and are not reproducible. The paper is also very hard to read: dense terminology (release pinning, quarantine bits, certificates, write tokens) buries the core idea, while key experimental details (edit statistics, extraction accuracy, cost) are missing.
Questions for authors
Clarifications:
- Does any SkillsBench information (task text, verifiers, outcomes, "configured task cases") enter evolution or validation? Appendix G shows the Qwen run evolved with qwen3.8-max; was the Opus column's graph evolved by Opus?
- What is the performance of a Qwen-evolved graph evaluated with Opus, and vice versa?
- Which edit types (e.g., type refinements like the light-curve example) account for the newly solved tasks?
- What is the precision of extracted Enables/CanFeed relations on a human-labeled sample?
- Cost (tokens, wall time) of evolution?
- Ablation: evolution without the source-consistency check and review gates.
- For ALFWorld, how were the ALFWorld-specific skills created, and how many are there? Suggestions: position the work against STRIPS/PDDL-style LLM planning more explicitly, since precondition/effect matching is classical. Move proof and system details to the appendix and bring evaluation details forward.
Score sensitivity: I could move to 3 if leakage is ruled out and evolution and governance are analyzed and ablated.
5 comments
Response to Reviewer KHvN (2/2)
§2 (cont.). Which edits matter (Q3) and edge precision (Q4)
- Newly solved tasks trace to specific edits (Appendix E, Tables 9 and 12). Evolution changes the top-5 on 58/72 (80.6%) Qwen and 55/72 (76.4%) Opus tasks (membership changes on 47 and 46), and target/artifact types make up 59.4% / 65.5% of published edits. Across all newly solved task-runs (45 on Qwen, 41 on Opus over 3 runs), 29 / 29 change the top-5 skill set and 7 / 2 reorder the same five: specifically, 13 per model bring in a skill absent from the base () candidate pool via an evolved edge (13/13 Qwen and 12/13 Opus paths use target-type refinements), and 23 / 18 reselect or reorder within the base () candidate pool; the remaining 9 / 10 keep the same top-5 while receiving newly compiled relation witnesses alongside the loaded skills.
- Example. In
earthquake-plate-calculation(Opus), the round-24 edit addinggeospatial_datasetunderdatasetretypes the output of the top BM25 seedgeospatial-analysison re-extraction; recompilation derivesgeospatial-analysis --CanFeed--> custom-distance-metrics, which enters the top-5 at rank 3, and the task is newly solved in runs 2 and 3. Appendix E adds anEnablesexample indrone-planning-control. - Edge precision (Q4). Human audit of 150 stratified edges per evolved graph (120
CanFeed, 30Enables) against the skill texts gives 81.3% precision on Qwen (80.8%CanFeed, 83.3%Enables) and 84.7% on Opus (82.5% / 93.3%). Remaining errors come mainly from coarse shared types in peripheral domains; this directly measures the extraction accuracy placed outside the Lean TCB in Section 3.4 (Table 1).
§4. Evaluation controls and statistics (W2, Q7)
- The LLM selector does not explain the gain. On SkillsBench, base SkillWorld () uses the same LLM selector, budget, pool, and harness as governed SkillWorld, yet stays near the retrieval baselines (35.65% / 53.70%) while domain types remain collapsed into
dataset; governed evolution adds +18.52 / +12.50 pp. On ALFWorld (frozen ontology), graph-based runs use no separate LLM reranker. - Exact means and SDs. Tables 2 and 4 now report exact unrounded 3-run means and sample SDs instead of whole-task rounded means (e.g., Opus governed is 47.67/72 = 66.20%; internal services is 26.33/36 = 73.1%), and Appendix E (Table 11) lists SkillsBench per-run counts. Governed SkillWorld's runs (39/40/38 on Qwen; 47/46/50 on Opus) never overlap with any run of any non-oracle baseline (best single baseline runs: 29 and 42).
- BM25 vs. BM25 + LLM reranking. Exact means are 25.00 vs. 24.67/72 on Qwen and 37.00 vs. 37.00/72 on Opus (runs 38/36/37 vs. 34/39/38); reranking wins 6 and loses 7 tasks on Qwen, and wins 7 and loses 7 on Opus. Within top-20 lexical hits, reranking only swaps borderline candidates, whereas typed
CanFeed/Enablesedges bring in non-lexical prerequisites. - Game-level paired tests on ALFWorld (Appendix F, Table 15). Two-sided sign-flip tests over the 134 unseen games (10,000 permutations, Holm–Bonferroni adjusted over 16 comparisons) show significant SR@15 gains for
Fullover No skills, graph-free search, and GoS on both models (adjusted ), and over BM25 and No relation labels on Qwen (adjusted ; unadjusted on DeepSeek, with 68.5%–72.4% fewer invalid actions). Against online-updating SkillDAG, frozenFullis +3.98 / +1.49 pp higher in SR@15 (unadjusted ; not significant) while cutting invalid actions by 49.7% / 70.0% (74 vs. 147; 51 vs. 170) without online graph updates. - ALFWorld skills (Q7). We corrected Section 4.2: ALFWorld uses GoS's public
skills_500library unchanged—37 ALFWorld procedural skills from Liu et al. (2026), not derived from our runs, plus 463 general/external distractors. - Internal-service tasks. In this proprietary deployment case study, the 36 tasks were curated from trajectories where skill-free execution struggled—using no skill-based method's outcome—and all baselines (including graph-free search and SkillDAG) run under the same harness on the shared 903-skill library.
§5. Positioning and readability. Section 5 now relates typed manifests and Enables/ConflictsWith compilation to STRIPS/PDDL and LLM-PDDL planning (e.g., Fikes & Nilsson, 1971; Ghallab et al., 2004; Silver et al., 2024), contrasting open-ended skill documents and read-only projection with closed-world planners; proof and system details moved to Appendices A–D so evaluation details appear earlier.
We believe these results meet the conditions in your score-sensitivity note, and we would be grateful if you would consider updating your score.
Follow-up on our response to Reviewer KHvN
Dear Reviewer KHvN,
As the discussion period wraps up soon, we wanted to briefly check in to see whether our response and revised manuscript resolve your questions—particularly on the three points highlighted in your review:
- No benchmark leakage: Clarified that offline evolution reads only static skill texts, manifests, and compiler diagnostics (
configured task casesare offline file-hash and edge checks), terminating automatically without checkpoint selection. - Evolution analysis: Added the edit taxonomy, provenance of newly solved tasks, human-audited edge precision (81.3% / 84.7%), and cross-model transfer (+9.72 / +8.33 pp without re-evolution).
- Governance ablation & statistics: Disabling the gates spends 1.28× the offline tokens within 4 rounds yet reaches only 37.04% on Qwen (vs. 54.17% with governance), alongside exact 3-run statistics and ALFWorld paired tests.
If any questions remain before the window closes, we would be very glad to clarify. Thank you again for your thorough review!
Response to Reviewer KHvN (1/2)
We thank Reviewer KHvN for the thorough technical assessment and for stating clearly what would resolve the concerns. Your score-sensitivity note names three conditions; we address each with new evidence:
§1. Leakage and model pairing (W1, Q1)
configured task casesare offline structural checks, not benchmark tasks. In Appendix D (now clarified), this term means 500 per-round checks testing (i) that each manifest'ssource_hashmatches its frozen skill file and (ii) edge consistency before and after the update.qwen3.8-max(51 rounds, 379 edits), and the Claude Opus 4.8 column's graph by Claude Opus 4.8 (41 rounds, 168 edits). Cross-model transfer below shows the result is not tied to the evolving model.§2. What evolution learns and what governance contributes (W1, Q2–Q4, Q6)
image_content_state) breaks compilation. Evaluating the 79 compilable edits shows the exact failures the gates block: (i) schema-breaking edits like the round-4 crash; (ii) 18 of 79 edits fail the Lean-verifiedSourceConsistentcheck on replay due to conflicting source claims; (iii) compiled edges jump by 75.6% in four rounds—e.g., merging distinct tool prerequisites into a genericdependencies_installedstate adds at least 22 unsupportedEnablesedges from Python setup skills (setup-env,uv-package-manager) to external tools (ffmpeg,gh_cli,lean4). Success is 37.04 ± 2.12% on Qwen (−17.13 pp vs. governed, and below even the Opus-transferred graph) and 56.48 ± 2.89% under Opus (−5.56 pp vs. the governed Qwen-evolved graph under Opus, 62.04%). The governance gates—including Lean-verified source consistency—rather than LLM call volume, account for the task gain.§3. Cost of evolution (Q5; Section 4.4, Table 13). Initial extraction costs 6.66M / 4.57M tokens (one call per skill); governed evolution costs 14.40M / 8.22M tokens over 51 / 41 rounds (412 / 194 re-extractions), 1.8–2.2× one extraction pass. Lean checking takes 0.44 / 0.69 s and compilation 2.32 s per 500-skill release on an Apple M4 Pro, without LLM calls.