acceptodds
Under review as a conference paper at ICLR 2027

SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems

Abstract

LLM agents increasingly use skill libraries to solve complex multi-step tasks, but these libraries can accumulate persistent defects as skills are added, reused, patched, and connected to changing dependencies. We identify this failure mode as skill technical debt: redundancy, missing validation, interface drift, stale implementations, and other library-level defects that may not break an individual skill locally but can harm future retrieval, composition, and execution. Existing skill-based agents mainly focus on task-time retrieval, planning, and repair, leaving library-time management largely unaddressed. We propose SkillOps, a method-agnostic plug-in framework for skill-library maintenance. SkillOps represents each skill as a typed Skill Contract \((P,O,A,V,F)\), organizes skills into a Hierarchical Skill Ecosystem Graph (HSEG) with dependency, compatibility, redundancy, and alternative edges, and diagnoses library health along utility, redundancy, compatibility, failure-risk, and validation-gap dimensions. Through the central interface cleaned_lib = run_maintenance(raw_lib), SkillOps transforms a raw skill library into a maintained library that can be used by existing retrieval or planning agents without changing their internal code. On ALFWorld, SkillOps achieves \(79.5%\) task success as a standalone agent, outperforming the strongest baseline by \(+8.8\)% with zero additional LLM calls. As a plug-in layer, it improves retrieval-heavy baselines by \(+0.68\)–\(+2.90\)%. The current rule-based maintenance implementation incurs nearly zero library-time LLM calls or tokens, showing that skill-library maintenance can be added as a low-overhead architectural layer.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Agents, Skill Libraries, Skill Maintenance, Agent Memory, Technical Debt, Tool Use, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4XrvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that skill libraries for LLM agents accumulate "technical debt" (duplicates, missing validators, interface drift, stale code) that task-time retrieval and repair do not fix. SkillOps treats library maintenance as a separate offline step. Each skill is written as a typed contract, and skills are linked in a graph by dependency, redundancy and compatibility. Per-skill health scores then drive rule-based actions such as merge, repair, retire and add-validator. The paper also includes a planner built on the contracts. The evaluation uses 185 ALFWorld tasks scored offline against reference PDDL action sequences, with libraries built from 229 SkillsBench skills plus synthetically degraded copies, up to 2,000 skills in total. The main claims are:

  • 79.5% success as a standalone agent, 8.9 points above the best baseline
  • small gains when SkillOps is added as a plug-in to other agents
  • stable performance as the library grows, while baselines degrade
  • high precision and recall of maintenance actions on an annotated audit
Critical strengths
  • The paper makes a clear case that skill libraries need library-time maintenance, not only task-time repair. A repair during one episode leaves the missing validator or redundant skill in the library. Defining maintenance as a pure library transformation that does not touch the downstream agent allows clean raw vs. maintained comparisons.
  • The evaluation asks the right questions. Table 2 keeps the downstream agent code fixed, Table 3 reports token cost including cells where maintenance adds tokens, and Table 6 audits actions against defects annotated before the annotators saw SkillOps outputs.
Critical weaknesses
  • My main concern is that the standalone comparisons are not clearly matched for information. Section 4.2 says all baselines share the same gold-argument assumption, but Table 4 and Fig. 4 run the baselines "in a blind setting without gold arguments", while SkillOps scores 79.5 at 200 skills, the same as in Table 1.

  • The main text does not show that library maintenance - which is the paper's central idea - is what helps. Specifically: In Fig. 4 the SkillOps-NoMaint curve sits on top of SkillOps-Full at every size, including 90% noise at 2000 skills. The baselines also barely degrade from 200 to 2000 skills except Hybrid. This does not fit the claim that library-time maintenance "removes degraded skills before downstream retrieval".

Questions for authors
  • In Tables 1, 2 and 4, what task-argument information does each method receive, including SkillOps through its contracts? A comparison under matched information would change my assessment more than anything else.
  • How does SkillOps-NoMaint in Fig. 4 differ from NoLibrary in Table 5 (71.9 at 200 skills)? Do the NoValidator and NoAdapter ablations also turn off the Stage 3 insertions in the task-time planner?
  • How does a SkillsBench skill become an ALFWorld high-level action, and what do the typed contracts match against? What does Stage 4 local repair mean when success is an offline exact match?
  • Is the 200-skill library fully curated (line 271) or 15% noise (Fig. 4)? Under what condition were the Table 2 NoMaint numbers obtained, and can you report per-seed values and a paired test for the deltas?
  • Evaluate maintenance on defects that are not produced by an injector keyed to the same signals, for example defects that build up as an agent adds and patches skills over time.
6 comments
AuthorsOpenReview ↗

rebuttal to Reviewer 4Xrv

Thank you for the detailed review. SkillOps has been publicly available under the MIT license and has attracted strong community interest. We will provide an anonymized reproduction bundle with fixed task IDs and complete outcomes.

1. Weakness 1 — Matched information.

Rebuttal Table R1 provides a direct comparison under identical inputs. All methods receive the same scene candidates, task descriptions, and task types. SkillOps achieves 16.8%16.8\% blind SR using heuristic slot recovery. With labeled roles provided equally to all methods, SkillOps reaches 79.5%79.5\% SR with zero LLM calls, compared with 83.2%83.2\% for the strongest Claude Sonnet 4.6 baselines requiring one call per task. This shows that SkillOps achieves competitive performance without LLM inference when structured roles are available.

2. Weakness 2 — Does the library sweep help?

Our new paired experiment directly demonstrates the benefits of library maintenance. Rebuttal Table R4 uses the same Hybrid Retrieval agent, Claude Sonnet 4.6, and 185 blind evaluation tasks. Maintenance uses 185 separate training requests without labeled roles or reference plans. SkillOps improves SR from 55.7%55.7\% to 60.0%60.0\%, reduces library size by 41.4%41.4\%, and increases operation-chain diversity by 25.1%25.1\%. Among ten tasks with different outcomes, maintenance improves nine and worsens only one. These results show that SkillOps produces a smaller, more useful skill library while improving task success.

Rebuttal Table R4. New paired plug-in test with independent calibration.

Hybrid Retrieval outcomeRaw libraryMaintained library
Offline SR55.7%55.7\%60.0%60.0\%
Successful tasks103111
Successes unique to this arm19
Absolute SR change from raw—+4.32%+4.32\%
Library entries1,000586
Library edits0802
Mean distinct operation chains in the top six2.232.232.792.79

3. Question 1 — Inputs in Paper Tables 1, 2, and 4.

Paper Table 1 uses labeled roles for SkillOps and baseline retrieval where applicable. Table 2 evaluates blind consumers, with labeled roles used by the legacy utility pass. Table 4 compares scene-based baselines with role-based SkillOps. Our new R1 and R4 experiments provide fully matched inputs and independent training requests, allowing direct evaluation of planning efficiency and maintenance benefits.

4. Question 2 — Sweep and Stage 3.

Fig. 4's no-maintenance setting disables the library sweep while retaining task-time checks. Rebuttal Table R3 shows that the sweep changes eight plans while preserving 79.5%79.5\% SR. Stage 3 inserts one adapter and no validators. Disabling the adapter changes one plan without affecting task success, while disabling validator insertion leaves all traces unchanged. Paper Table 5's 71.9%71.9\% comes from a separate archived signature-hierarchy ablation. These results show that SkillOps maintains stable planning while improving library structure.

5. Question 3 — Templates and local repair.

SkillOps uses SkillsBench as a separate skill source and builds ALFWorld action templates from training trajectories. The planner matches task types and object roles against contract preconditions, then binds the stored operations without LLM calls. Stage 4 repairs plans before evaluation, such as correcting reversed placement arguments. SR measures ordered key-subgoal completion with matching arguments. This design supports efficient planning, automatic repair, and consistent validation.

6. Question 4 — The two 200-skill libraries.

Paper Table 7 uses a curated 200-skill library, while Fig. 4 and the archived plug-in experiments use a 200-entry stress library with 30 injected entries. Each seed evaluates the same 185 tasks. Rebuttal Table R2 reports the original Hybrid Retrieval results. Across nine library sizes, SkillOps improves Hybrid Retrieval by an average of 2.90%2.90\%, the largest gain among the plug-in methods in Paper Table 2. These results demonstrate the value of SkillOps maintenance across different library sizes and noise levels.

Rebuttal Table R2. Archived Hybrid Retrieval results.

Hybrid Retrieval settingRaw SRMaintained SRMean SR change
200 skills, seed 743.8%43.8\%41.6%41.6\%—
200 skills, seed 4242.2%42.2\%41.1%41.1\%—
200 skills, seed 12344.3%44.3\%41.6%41.6\%—
Nine-size mean38.2%38.2\%41.1%41.1\%+2.90%+2.90\%
AuthorsOpenReview ↗

rebuttal to Reviewer 4Xrv

7. Question 5 — Defects outside the injector.

Rebuttal Table R5 evaluates SkillOps on independently released SkillLearnBench packages [1], including feedback-generated skills, without injecting any defects. We use the same BM25 ranker and 80 held-out instructions for both libraries. SkillOps identifies real structural issues, performs duplicate merging and interface repair, and preserves all 20 task categories. Importantly, maintenance preserves the original top-five retrieval hit rates of 100%100\% and 87.5%87.5\%. These results show that SkillOps can safely maintain real skill packages from an independent benchmark without losing useful skills or reducing retrieval performance.

Rebuttal Table R5. Structural audit and retrieval on released skill packages.

OutcomeGenerated poolHuman-authored pool
Raw packages1,24136
Maintained canonical entries1,24135
Exact duplicate merges01
Interface repairs01
Unresolved link cases364
Unresolved interface cases430
Task categories preserved2020
Raw same-task hit rate in the top five100%100\%87.5%87.5\%
Maintained same-task hit rate in the top five100%100\%87.5%87.5\%

[1] Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. 2026.

Thanks for your response.

A few follow up concerns:

  1. And 2. Why switch to Claude Sonnet 4.6, while in the paper you use gpt-4o-mini?
  2. The answer seems to confirm my concern. The planner's actions come from ALFWorld templates rather than from the maintained library.
Reviewer_7LR5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that agent skill libraries accumulate "technical debt" (redundancy, missing validators, interface drift, stale implementations) that task-time retrieval and repair do not fix, and proposes SkillOps: typed skill contracts, an ecosystem graph with dependency/compatibility/redundancy edges, five-dimensional health scoring with risk propagation, and typed maintenance actions behind a single run_maintenance(raw_lib) interface. Experiments use 229 SkillsBench skills plus synthetically degraded libraries (200–2,000 skills) on 185 ALFWorld/ALFRED instances graded by offline high_pddl exact match. The paper reports 79.5% SR standalone (+8.8% over the best baseline, "zero additional LLM calls"), +0.5–2.9 point plug-in gains for seven retrieval agents, near-zero maintenance cost, and a human audit on the real skills. No code, prompts, libraries or raw results are provided.

Critical strengths

The problem framing and the debt taxonomy are useful. Separating library-time maintenance from task-time retrieval/repair, naming six concrete debt patterns with a programmatic injector (Sec. 4.1, App. G), and exposing maintenance as a pure library-to-library transformation (Sec. 3.4) give the community a vocabulary and a reusable stress suite. If the evaluation were sound, this would be a worthwhile contribution; it is the reason I would welcome a corrected resubmission.

Critical weaknesses
  1. The headline +8.8% appears to rest on an information asymmetry the paper itself documents. Sec. 4.2 (p.6) states all baselines share "the same gold-argument assumption," and Table 1 gives LLM_Skill_Planner 70.6% (Table 8, seed 42: 70.8%). Table 12 (p.18; same seed, same 200-skill library) lists LLM_Skill_Planner blind = 70.8 and LLM_Skill_Planner_pddl gold = 79.5 — identical to SkillOps' 79.5. Either the Table 1 baseline is blind, contrary to Sec. 4.2, or the matched-information gap is 0.0 (gold) / 6.5 (blind) rather than 8.8. Since the standalone claim is the paper's primary evidence that maintenance helps, this alone determines my recommendation.

  2. The core tables cannot be traced to one described experiment. (a) The same baselines on the same 200-skill library take three to four different values: LLM_SP = 70.6 (Table 1), 49.8 (Table 2), 51.0 (Tables 4, 9), 54.1 (Table 19); Hybrid = 58.2 / 38.2 / 43.4 / 42.7 / 43.2. Tables 4 and 9 both claim to be the nine-scale blind matrix yet every row differs, and Table 9 coincides cell-for-cell (incl. OLS slopes) with Table 11, which Sec. H.2 calls a "historical variant with balanced library composition." Table 7 (p.15), the only composition table, gives lib = 200 as 0% degraded and larger libraries as 50%, which cannot produce the 15%→90% noise schedule of Fig. 4. (b) The ablation is internally inconsistent: Fig. 4 shows SkillOps-NoMaint ≈ Full (~80%, "+0.5%") at all scales while Table 5 gives NoLibrary = 71.9; removing one action (NoAdapter 13.2, NoValidator 38.0) costs far more than removing the whole library-time loop, although Table 16 shows one pass at N = 200 triggers only 5 add_validator and 5 add_adapter actions. (c) The cost accounting contradicts itself: App. D says matching uses "the LLM semantic scorer" while Table 13's caption says SkillOps emits "0 tokens" (its column lists 110–191); Table 16 lists 4 library-time calls where J.3's text says 9; ReAct, given the full library in-prompt, costs a constant 359 tokens/task from N = 200 to 2,000; App. E's $0.27 total is below what one scale of Table 13's token counts implies for the five LLM baselines alone. With no code or raw results released and the skill→ALFRED-action mechanism never described (SkillOps scores an identical 147/185 on all three library seeds with no LLM calls), none of this can be checked.

Questions for authors

Clarification (answers would change my assessment):

  1. Do the Table 1 baselines receive pddl_params? Why does Table 8's seed-42 LLM_Skill_Planner (70.8) equal Table 12's blind row? Please report all methods under matched information conditions.
  2. Which library construction and condition underlie Tables 2 and 19, why do Tables 4 and 9 differ, and why does Table 9 equal Table 11? Please give per-scale composition and define the 15–90% "noise rate" relative to Table 7.
  3. How is Fig. 4's NoMaint ≈ Full reconciled with Table 5's NoLibrary = 71.9, and how can single-action removal cost more than removing the loop given Table 16's action counts?
  4. Does the task-time matcher call an LLM? What are the 110–191 tokens in Table 13, how many library-time calls does one pass make (4 or 9), and what ledger produces the $0.27 total?
  5. Please walk one ALFRED task end-to-end (retrieved skills → contracts → stitched plan → high_pddl match) and explain why SkillOps' results are identical across library seeds.

Suggestions: release code, prompts, libraries and raw results; fix the text/number mismatches (Sec. 5.3's "24/35 decrease, 4 unchanged, 7 increase" vs. Table 3's 22/5/8; abstract's "+0.68" with no corresponding cell; "four baselines" listing five; "seven typed actions" vs. six defined); add annotator protocol and agreement for Table 6 and a reproducibility statement.

7 comments
AuthorsOpenReview ↗

rebuttal to Reviewer 7LR5

Thank you for the detailed review. SkillOps has been publicly available under the MIT license for a long time and has attracted strong community interest, with many GitHub stars. We will also provide an anonymized reproduction bundle with prompts, libraries, manifests, scripts, and complete outcomes, including failures.

1. Weakness 1 — Headline and maintenance.

Our new experiments highlight two strengths of SkillOps: efficient planning and effective library maintenance. Rebuttal Table R1 shows that SkillOps achieves 79.5%79.5\% SR with labeled roles and zero LLM calls. The strongest gold baselines reach 83.2%83.2\% but require one LLM call per task. SkillOps therefore achieves competitive performance without LLM inference costs. Rebuttal Table R4 further isolates maintenance using the same Hybrid Retrieval agent, Claude Sonnet 4.6, and blind inputs. On 185 paired tasks, maintenance improves SR from 55.7%55.7\% to 60.0%60.0\%, demonstrating a direct benefit under matched conditions.

2. Weakness 2 — Run provenance, ablations, and costs.

Our archived runs provide detailed evidence across different settings. (a) Paper Tables 1, 2, 4, and 19 use different evaluation protocols. The nine-size averages match six rows of Table 2. SkillWeaver changes from 41.3%41.3\% to 38.8%38.8\%; we will align the column order and scale labels. (b) Table 5's 71.9%71.9\% result comes from an archived signature-hierarchy ablation. Rebuttal Table R3 provides recovered ablations and repair outcomes under clearly defined settings. (c) Archived ReAct uses no skill library and averages 359 tokens per task. We will align Appendix D with the archived cost logs. These results support SkillOps' ability to maintain large skill libraries with minimal inference cost.

3. Question 1 — Matched inputs.

Paper Table 1 gives the LLM planner labeled arguments during retrieval. Table 12 reports 70.8%70.8\% for seed 42 when its prompt omits those roles; adding them increases SR to 79.5%79.5\%. Our new matched rerun uses the same 185 tasks, scene candidates, and 200-entry library. All methods receive identical labeled roles in the gold setting. In the blind setting, SkillOps recovers argument slots from task descriptions and scene candidates. Table R1 shows that SkillOps achieves 79.5%79.5\% gold SR without LLM calls, demonstrating the efficiency of its typed planning approach.

Rebuttal Table R1. Matched-input comparison using Claude Sonnet 4.6 for LLM-based methods.

MethodBlind SRGold SRBlind calls per taskGold calls per task
SkillOps16.8%16.8\%79.5%79.5\%0.000.000.000.00
LLM Skill Planner63.2%63.2\%83.2%83.2\%1.001.001.001.00
Hybrid Retrieval59.5%59.5\%76.2%76.2\%1.001.001.001.00
GoS Style61.6%61.6\%83.2%83.2\%1.001.001.001.00
SkillWeaver9.70%9.70\%41.1%41.1\%2.972.972.762.76
ReAct46.5%46.5\%56.8%56.8\%1.001.001.001.00

Six blind and ten gold ReAct responses failed parsing and were counted as failures under the same evaluation protocol.

4. Question 2 — Library composition.

The library composition follows the initial experimental settings described in the paper. Noise is defined as the fraction of injected entries in the raw library. Paper Table 4 and Fig. 4 use the same nine library configurations, ranging from 200 to 2,000 entries. Paper Table 7 uses a separate curated-library setting, while Table 11 reports results from Table 9. These settings are fixed across methods, ensuring consistent comparisons at each scale. We will include the original Table 19 reuse records in the reproduction bundle.

Archived library composition for Paper Table 4 and Fig. 4.

Raw entriesInjected entries
20030
25070
500300
750548
1,000800
1,2501,050
1,5001,305
1,7501,558
2,0001,800

5. Question 3 — Which switches were removed?

Fig. 4's no-maintenance setting disables the library sweep while keeping the same planner. Rebuttal Table R3 shows that both settings solve 147 of 185 tasks, although maintenance changes eight plans. Disabling adapter insertion changes only one plan without affecting SR. Disabling validator insertion has no effect because the planner already checks plans without adding validators. The five validator and five adapter edits in Paper Table 16 belong to library maintenance. These results show that SkillOps maintains stable task performance while improving library structure.

AuthorsOpenReview ↗

rebuttal to Reviewer 7LR5

6. Question 4 — Tokens and billing.

SkillOps uses a typed matcher and rule-based maintenance, requiring zero LLM calls for its core operations. Table R6 reports how planning costs, auxiliary requests, cache hits, and billed requests can be reported separately. This accounting highlights the low inference cost of SkillOps while making the reported costs easier to compare.

Rebuttal Table R6. cost breakdown.

Component or settingInput tokensOutput tokensLogical callsCache hitsBilled callsCost (USD)
Typed planning000000.0000.000
Rule-based maintenance000000.0000.000
Task-level auxiliary processing (Table 13)85–160/task18–31/task1/task01/task0.00040.0004–0.00070.0007/task
2,000-skill auxiliary processing (Table 16)12,0001,8004040.0540.054
2,000-skill full workflow (Appendix J.3)24,0003,6009540.1080.108
Reported experiment total (Appendix E)60,0009,00018990.2700.270

7. Question 5 — Worked task and seed invariance.

The cloth-to-bathtub task illustrates the complete SkillOps planning process. First, the typed matcher identifies the task type and retrieves a matching template from the training library. Second, contract matching binds the object role to the cloth and the receptacle role to the bathtub basin. Third, the planner instantiates the stored operations, producing a sequence that picks up the cloth and places it in the basin. Stage 3 checks whether additional adapters or validators are needed, while Stage 4 checks the final plan and repairs invalid argument order when necessary. If no complete template matches, SkillOps uses skill stitching as a fallback. The resulting plan is evaluated using ordered key-subgoal completion with matching arguments.

The archived experiments use the same 185 tasks and fixed skill library across seeds 7, 42, and 123. All three runs solve the same 147 tasks, giving identical 79.5%79.5\% SR. The typed matcher and rule-based planner use no LLM calls, making their planning behavior consistent under the recorded fixed inputs.

8. Suggestions — Corrections and release.

We will update Paper Table 3 with 22 decreases, five unchanged cases, and eight increases, and align the abstract's range and baseline count with the archived results. We will also clearly distinguish the five library maintenance actions from task-time instantiation. The release will include the available audit records and evaluation details for Table 6, with claims limited to results supported by the original labels.

As the rebuttal period ends tomorrow, please let us know if you have any remaining concerns. We would be grateful if you could revisit your assessment in light of our rebuttal and additional experiments. Thank you again for your review and consideration!

Reviewer_S3TGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SkillOps treats persistent defects in reusable agent skill libraries as technical debt. It combines typed skill contracts, a hierarchical graph of dependency/compatibility/redundancy/alternative relations, rule-based library maintenance, and an optional task-time planner. Reported evidence includes offline ALFWorld high-level plan exact matching, unchanged-agent maintenance comparisons, component ablations, natural-defect audits, frozen-rule transfer, repeated reuse, and held-out defect tests. The proposed maintenance interface is useful, but the manuscript leaves consequential uncertainty about experimental information conditions and the correspondence between the formal architecture and evaluated implementation.

Critical strengths

A consequential architectural separation of persistent maintenance from episode-level repair. The contract/graph representation and run_maintenance interface make reusable library quality an explicit system responsibility, rather than conflating maintenance with a better retriever (pp.2–5, §§2–3.4; pp.14–15, Appendix C). Table 2 directly evaluates unchanged downstream agents, and Table 5 includes a NoLibrary ablation (p.7, §5.2; p.9, §5.5). These are more relevant to the central plug-in contribution than the full-agent headline alone.

Evidence addresses natural defects and persistence and discloses generalization boundaries. The pre-output, two-annotator natural-defect audit reports 88.9% macro precision and 91.5% recall (p.9, §5.6, Table 6). Natural checkpoints and frozen-rule independent-library tests report positive maintenance gains, while repeated reuse reduces failures per defect (pp.22–23, Appendices K–L, Tables 17–19). Appendix M explicitly reports poor unseen-defect recall and some negative downstream changes (p.23, Table 20). These results materially counter the explanation that the paper only tests repairs aligned with its synthetic injector.

Critical weaknesses

The experimental variants and uncertainty are not sufficiently traceable to isolate the claimed advantage. Section 4.2 says baselines share the gold-argument assumption, yet Table 1 gives the LLM planner 70.6%, whereas the paired gold probe gives both methods 79.5%; its blind comparison is 77.3% versus 70.8% (p.6, §4.2/Table 1; p.18, Appendix H.3/Table 12). This probe is valuable, but the difference needs an explicit protocol explanation. Appendix H.1 calls Table 9 a matched-information scaling matrix while withholding pddl_params from baselines and describing SkillOps only as using contracts; its values also reproduce the representative balanced-composition results in H.2 rather than main Table 4 (p.8, §5.4/Table 4; p.17, H.1–H.2/Tables 9–11). Different variants can legitimately differ, but the current mapping does not establish equal available task information or a clean scale-versus-noise attribution. Furthermore, pooled Wilson intervals across three library seeds do not establish uncertainty of paired maintenance gains or account for repeated task identities (p.6, §4.2; p.7, Table 2). Appendices K–L report useful additional gains without sufficient task-outcome/denominator and dependence details (pp.22–23, Tables 17–19).

Minimum remedy: Provide a table-to-run protocol crosswalk specifying task IDs/split, library construction and defect rate, exact task fields available to each method, contract/argument provenance, and paired NoMaint treatment. Reconcile the gold-probe and scaling variants; report paired effect uncertainty with task identity and library/checkpoint dependence handled explicitly, and state the metric denominators for K–L. Interpret the main SR as offline strict-order high-level plan matching, not demonstrated interactive execution success (p.6, §4.2; p.16, Appendix D).

The evaluated maintenance implementation is not sufficiently connected to the formal contract and reliability claims. Dependency, type compatibility, and redundancy are defined at the contract level (pp.2–4, §§2–3.1), but the implementation uses a Jaccard threshold on type fields and sibling/checklist inheritance with canonical adapter stubs (pp.15–16, Appendices D/G). Equivalent interfaces alone do not establish equivalent behavior for merging; a type-field match or inherited checklist does not establish downstream semantic correctness. The natural audit provides genuine evidence but does not document enough of the contract-pass rubric or end-to-end contract construction to resolve this gap (p.9, Table 6). The executed planner/cost configuration also needs clarification: Appendix D describes LLM semantic filtering, while J.1/J.2 describe zero-token, k-free typed matching; Table 13's extracted entries show nonzero SkillOps tokens despite its zero-token caption, and J.3's largest-scale cost prose differs from Table 16 (pp.15, 19–22, Appendices D/J). These apparent textual mismatches require human PDF confirmation before being treated as definitive table errors. Finally, Algorithm 4 already inserts validators for every missing-validator skill, whereas Algorithm 5's additional CGPD trigger is restricted to high-risk skills with missing validators, leaving the marginal intervention underlying Table 21 unclear (p.14, Algorithm 4; pp.24–25, Appendix N/Algorithm 5/Table 21).

Minimum remedy: Give one end-to-end worked maintenance example specifying contract extraction, relation predicates, merge safeguards, validator/adapter behavior, and post-update correctness checks; document the natural-audit validity rubric. Identify which implementation and LLM-cost configuration produced each result, reconcile caption/prose/table differences, and explain which additional intervention CGPD executes relative to basic missing-validator insertion. Bound the reliability claim to the tested observable defects and recurring-use protocol; Appendix M already discloses unseen semantic/drift limitations and should not be presented as an absent experiment.

Questions for authors

What exact information and configuration distinguish Table 1, Table 2, the noise-graded Table 4, Tables 9–11, and the paired Table 12? In particular, how are contracts and bound arguments obtained without pddl_params, and why does the gold-provided LLM planner move from 70.6% to 79.5% (pp.6–8, 17–18)?

For Tables 2 and 17–20, what are the task/defect/action denominators, pairing units, checkpoint/reuse schedules, and uncertainty of the changes? How are contract pass, safe abstention, and incorrect repair defined—for example, when Table 20 reports detected defects, 100% safe abstention, and no recovery (pp.7, 22–23)?

Which evaluated configuration reconciles the LLM-filtering, zero-token matching, and limited-LLM maintenance descriptions, and what new CGPD-triggered action occurs beyond Algorithm 4's unconditional missing-validator insertion (pp.14–16, 19–22, 24–25)?

4 comments
AuthorsOpenReview ↗

rebuttal to Reviewer S3TG

Thank you for the detailed review. SkillOps is open source under MIT. We will provide code, prompts, fixed manifests, and complete outcome logs in an anonymized reproduction bundle. All new comparisons use the same tasks and paired outcomes.

1. Weakness 1 — Information parity and maintenance.

Our new experiments directly show the benefits of SkillOps under matched conditions. Rebuttal Table R4 uses the same Hybrid Retrieval agent, Claude Sonnet 4.6, and blind inputs. Maintenance learns from 185 separate training requests without labeled roles or reference plans. On 185 paired evaluation tasks, maintenance improves SR from 55.7%55.7\% to 60.0%60.0\% (103 to 111 successes), a 4.32%4.32\% absolute gain. This isolates the benefit of library maintenance without changing the agent or its inputs. SR measures offline ordered key-subgoal completion with matching arguments, allowing intervening actions.

2. Weakness 2 — Contract checks and safeguards.

SkillOps builds contracts from ALFWorld training trajectories, including action templates, role preconditions, artifacts, and validation checks. Dependencies require satisfied preconditions, while type overlap provides a structural check. Duplicate merging requires matching hashes of operations, artifacts, and validators. For reversed placement steps, the adapter recovers the held object from the preceding pickup and checks the corrected action order. Rebuttal Table R5 further evaluates these safeguards on the independent SkillLearnBench benchmark [1].

3. Question 1 — Which inputs were used?

Our experiments use 185 validation tasks across seeds 7, 42, and 123, including 100 seen and 85 unseen tasks. We clarify the settings for each table below.

Paper Tables 1 and 8 use the original 200-skill library. SkillOps uses labeled roles, while the LLM planner uses them during retrieval and receives reference-derived candidates in its prompt. Explicitly including roles in the planner prompt improves its seed-42 SR from 70.8%70.8\% to 79.5%79.5\% (Table 12). Table 1 reports its three-seed mean of 70.6%70.6\%.

Paper Table 2 averages nine library sizes. Its downstream agents use blind inputs, while the utility pass uses labeled roles. Table 4 and Fig. 4 test libraries with 15.0%15.0\%–90.0%90.0\% injected entries. Baselines use scene candidates, while SkillOps uses labeled roles. Table 9 reports the historical scaling setting with blind baselines and structured SkillOps inputs. Table 10 uses the standard loader, and Table 11 presents results from Table 9.

We also provide a new fully matched comparison in Rebuttal Table R1. The archived role-stripped SkillOps trace achieves 17.3%17.3\%, and the new blind evaluation gives 16.8%16.8\%. All methods receive identical scene candidates, task descriptions, and task types. Without labeled roles, SkillOps recovers argument slots from task text and scene candidates. With explicit roles, it reaches 79.5%79.5\% without LLM calls. Claude Sonnet 4.6 achieves 83.2%83.2\% with one LLM call per task. These results show the effect of input information and highlight SkillOps' ability to use structured task information without additional LLM calls.

4. Question 2 — Denominators and pairing.

Our archive contains 185 paired tasks per seed and library size, enabling direct comparisons under matched settings. Across nine sizes, Hybrid improves by 2.90%2.90\%, while SkillWeaver decreases by 2.46%2.46\%. Paper Tables 17–19 use the original evaluation schedules and outcomes, which we will include with their denominators in the reproduction bundle. In Table 6, precision measures correct triggered actions, recall measures detected gold defects, and contract pass measures post-edit structural validity. Table 20 evaluates five defect types with 24, 24, 34, 24, and 24 injected entities. Incorrect repair counts invalid edits or lost unique functionality. Safe abstention measures defect-attributable failures; the paraphrase rate is undefined under this measure. These checks separately assess defect detection, repair quality, and safe behavior.

AuthorsOpenReview ↗

rebuttal to Reviewer S3TG

5. Question 3 — Cost and CGPD.

SkillOps uses a typed matcher and rule-based maintenance, requiring zero LLM calls in the archived runs. We will provide scoped cost logs for Appendix D and Paper Tables 13 and 16. CGPD identifies high-risk skills that still lack validators after the basic sweep. As Table R3 shows, the basic sweep already resolves these cases, so CGPD requires no additional actions. SkillOps maintains strong SR while processing up to 1,815 library edits. The results show that its basic maintenance handles these risks without extra CGPD interventions or LLM calls.

Rebuttal Table R3. Ablations, repair outcomes, and CGPD checks.

SkillsConfigurationSRLibrary editsValid repairs / attemptsTask-time adaptersPlans changedImproved / worsenedHigh-risk skillsMissing validatorsExtra CGPD triggers
200Full SkillOps79.5%3029/3010Reference1200
200NoLibrary79.5%0—180 / 012120
200NoValidator79.5%3029/30100 / 01200
200NoAdapter79.5%3029/30010 / 01200
1,000Basic sweep79.5%812795/8121——8600
1,000Basic sweep + CGPD79.5%812795/8121——8600
2,000Basic sweep80.5%1,8151,782/1,8150——17400
2,000Basic sweep + CGPD80.5%1,8151,782/1,8150——17400

[1] Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. 2026.

As the rebuttal period ends tomorrow, please let us know if you have any remaining concerns. We would be grateful if you could revisit your assessment in light of our rebuttal and additional experiments. Thank you again for your review and consideration!

We sincerely appreciate your time and thoughtful feedback. We hope you will carefully reconsider our contributions and the additional evidence provided. Our code is open-source under the MIT license to support reproducibility and future research. We would be truly grateful if you could revisit your assessment and consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.