- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that skill libraries for LLM agents accumulate "technical debt" (duplicates, missing validators, interface drift, stale code) that task-time retrieval and repair do not fix. SkillOps treats library maintenance as a separate offline step. Each skill is written as a typed contract, and skills are linked in a graph by dependency, redundancy and compatibility. Per-skill health scores then drive rule-based actions such as merge, repair, retire and add-validator. The paper also includes a planner built on the contracts. The evaluation uses 185 ALFWorld tasks scored offline against reference PDDL action sequences, with libraries built from 229 SkillsBench skills plus synthetically degraded copies, up to 2,000 skills in total. The main claims are:
- 79.5% success as a standalone agent, 8.9 points above the best baseline
- small gains when SkillOps is added as a plug-in to other agents
- stable performance as the library grows, while baselines degrade
- high precision and recall of maintenance actions on an annotated audit
Critical strengths
- The paper makes a clear case that skill libraries need library-time maintenance, not only task-time repair. A repair during one episode leaves the missing validator or redundant skill in the library. Defining maintenance as a pure library transformation that does not touch the downstream agent allows clean raw vs. maintained comparisons.
- The evaluation asks the right questions. Table 2 keeps the downstream agent code fixed, Table 3 reports token cost including cells where maintenance adds tokens, and Table 6 audits actions against defects annotated before the annotators saw SkillOps outputs.
Critical weaknesses
-
My main concern is that the standalone comparisons are not clearly matched for information. Section 4.2 says all baselines share the same gold-argument assumption, but Table 4 and Fig. 4 run the baselines "in a blind setting without gold arguments", while SkillOps scores 79.5 at 200 skills, the same as in Table 1.
-
The main text does not show that library maintenance - which is the paper's central idea - is what helps. Specifically: In Fig. 4 the SkillOps-NoMaint curve sits on top of SkillOps-Full at every size, including 90% noise at 2000 skills. The baselines also barely degrade from 200 to 2000 skills except Hybrid. This does not fit the claim that library-time maintenance "removes degraded skills before downstream retrieval".
Questions for authors
- In Tables 1, 2 and 4, what task-argument information does each method receive, including SkillOps through its contracts? A comparison under matched information would change my assessment more than anything else.
- How does SkillOps-NoMaint in Fig. 4 differ from NoLibrary in Table 5 (71.9 at 200 skills)? Do the NoValidator and NoAdapter ablations also turn off the Stage 3 insertions in the task-time planner?
- How does a SkillsBench skill become an ALFWorld high-level action, and what do the typed contracts match against? What does Stage 4 local repair mean when success is an offline exact match?
- Is the 200-skill library fully curated (line 271) or 15% noise (Fig. 4)? Under what condition were the Table 2 NoMaint numbers obtained, and can you report per-seed values and a paired test for the deltas?
- Evaluate maintenance on defects that are not produced by an injector keyed to the same signals, for example defects that build up as an agent adds and patches skills over time.
6 comments
rebuttal to Reviewer 4Xrv
7. Question 5 — Defects outside the injector.
Rebuttal Table R5 evaluates SkillOps on independently released SkillLearnBench packages [1], including feedback-generated skills, without injecting any defects. We use the same BM25 ranker and 80 held-out instructions for both libraries. SkillOps identifies real structural issues, performs duplicate merging and interface repair, and preserves all 20 task categories. Importantly, maintenance preserves the original top-five retrieval hit rates of and . These results show that SkillOps can safely maintain real skill packages from an independent benchmark without losing useful skills or reducing retrieval performance.
Rebuttal Table R5. Structural audit and retrieval on released skill packages.
| Outcome | Generated pool | Human-authored pool |
|---|---|---|
| Raw packages | 1,241 | 36 |
| Maintained canonical entries | 1,241 | 35 |
| Exact duplicate merges | 0 | 1 |
| Interface repairs | 0 | 1 |
| Unresolved link cases | 36 | 4 |
| Unresolved interface cases | 43 | 0 |
| Task categories preserved | 20 | 20 |
| Raw same-task hit rate in the top five | ||
| Maintained same-task hit rate in the top five |
[1] Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. 2026.
Thanks for your response.
A few follow up concerns:
- And 2. Why switch to Claude Sonnet 4.6, while in the paper you use gpt-4o-mini?
- The answer seems to confirm my concern. The planner's actions come from ALFWorld templates rather than from the maintained library.
rebuttal to Reviewer 4Xrv
Thank you for the detailed review. SkillOps has been publicly available under the MIT license and has attracted strong community interest. We will provide an anonymized reproduction bundle with fixed task IDs and complete outcomes.
1. Weakness 1 — Matched information.
Rebuttal Table R1 provides a direct comparison under identical inputs. All methods receive the same scene candidates, task descriptions, and task types. SkillOps achieves 16.8% blind SR using heuristic slot recovery. With labeled roles provided equally to all methods, SkillOps reaches 79.5% SR with zero LLM calls, compared with 83.2% for the strongest Claude Sonnet 4.6 baselines requiring one call per task. This shows that SkillOps achieves competitive performance without LLM inference when structured roles are available.
2. Weakness 2 — Does the library sweep help?
Our new paired experiment directly demonstrates the benefits of library maintenance. Rebuttal Table R4 uses the same Hybrid Retrieval agent, Claude Sonnet 4.6, and 185 blind evaluation tasks. Maintenance uses 185 separate training requests without labeled roles or reference plans. SkillOps improves SR from 55.7% to 60.0%, reduces library size by 41.4%, and increases operation-chain diversity by 25.1%. Among ten tasks with different outcomes, maintenance improves nine and worsens only one. These results show that SkillOps produces a smaller, more useful skill library while improving task success.
Rebuttal Table R4. New paired plug-in test with independent calibration.
3. Question 1 — Inputs in Paper Tables 1, 2, and 4.
Paper Table 1 uses labeled roles for SkillOps and baseline retrieval where applicable. Table 2 evaluates blind consumers, with labeled roles used by the legacy utility pass. Table 4 compares scene-based baselines with role-based SkillOps. Our new R1 and R4 experiments provide fully matched inputs and independent training requests, allowing direct evaluation of planning efficiency and maintenance benefits.
4. Question 2 — Sweep and Stage 3.
Fig. 4's no-maintenance setting disables the library sweep while retaining task-time checks. Rebuttal Table R3 shows that the sweep changes eight plans while preserving 79.5% SR. Stage 3 inserts one adapter and no validators. Disabling the adapter changes one plan without affecting task success, while disabling validator insertion leaves all traces unchanged. Paper Table 5's 71.9% comes from a separate archived signature-hierarchy ablation. These results show that SkillOps maintains stable planning while improving library structure.
5. Question 3 — Templates and local repair.
SkillOps uses SkillsBench as a separate skill source and builds ALFWorld action templates from training trajectories. The planner matches task types and object roles against contract preconditions, then binds the stored operations without LLM calls. Stage 4 repairs plans before evaluation, such as correcting reversed placement arguments. SR measures ordered key-subgoal completion with matching arguments. This design supports efficient planning, automatic repair, and consistent validation.
6. Question 4 — The two 200-skill libraries.
Paper Table 7 uses a curated 200-skill library, while Fig. 4 and the archived plug-in experiments use a 200-entry stress library with 30 injected entries. Each seed evaluates the same 185 tasks. Rebuttal Table R2 reports the original Hybrid Retrieval results. Across nine library sizes, SkillOps improves Hybrid Retrieval by an average of 2.90%, the largest gain among the plug-in methods in Paper Table 2. These results demonstrate the value of SkillOps maintenance across different library sizes and noise levels.
Rebuttal Table R2. Archived Hybrid Retrieval results.