- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SkillWeave studies how a frozen LLM agent can improve from accumulated conversational feedback without receiving reference answers or task-success labels during adaptation. It jointly maintains procedural skills and a knowledge base. Feedback logs are grouped into topics to create or revise skills, while factual gaps are addressed through evidence-grounded knowledge accumulation. Skills also include applicability conditions and optional knowledge-retrieval instructions. The main contribution is a concrete design for coordinating procedural and factual learning from mixed, partial feedback. The paper reports the highest MemoryBench average among the evaluated methods on each of three backbones and competitive results on ExpertLongBench.
Critical strengths
-
The paper offers useful, concrete methodological contributions. Distinguishing factual corrections, procedural lessons, and applicability feedback is a sensible response to conversational feedback that mixes these signals. Aggregating related feedback before revising a skill, retaining supporting knowledge, and recording when a skill should not apply form a coherent implementation. I see these choices as meaningful incremental contributions with practical value.
-
The empirical evaluation provides encouraging evidence of usefulness. The MemoryBench results cover several domains and three backbones, with additional evaluation on long-document tasks. The ablations and repeated accumulation runs help establish that the complete system is useful and that its reported gains are not based only on a single accumulation run. This makes the method worth further consideration, even though the experiments below are needed to identify what drives its advantage.
Critical weaknesses
-
The contribution needs clearer positioning within skill–memory co-evolution. At a conceptual level, SkillWeave coordinates the evolution and use of procedural skills and factual memory. The broad contrast that existing approaches maintain only one kind of store does not adequately place it within related approaches. For example, MSCE, Sections 3–4 develops memory and skills together, including environmental knowledge that informs skill use. Mem²Evolve, Section 3 couples experience memory with reusable tools and expert agents. These methods have different representations and feedback protocols, so they should not be treated as equivalent to SkillWeave. However, the paper needs to identify which new mechanism or capability follows from its conversational-feedback setting, beyond maintaining complementary stores. This affects how much novelty can be assigned to the framework itself, while leaving room for its specific designs to be valuable. (Lines 047–060 and 079–101.)
-
The current controls do not sufficiently isolate the value of the proposed coordination and topic aggregation. The knowledge-base ablation and raw-Wikipedia controls are useful, but they do not establish whether an existing skill learner given the same curated knowledge could obtain similar gains. Separately, removing clustering changes both the update procedure and the resulting skill library: the paper reports 36 skills versus 8 for the full method. The resulting performance gap therefore cannot be attributed specifically to grouping feedback by topic. These are fixable evidence gaps, but they concern the central explanation for why this method improves over simpler alternatives. (Table 2; lines 351–374.)
Questions for authors
-
What specifically distinguishes SkillWeave's skill–memory co-evolution from related approaches such as MSCE and Mem²Evolve? Please compare the stored representations, available feedback, use of success signals, knowledge sources, and dependencies between skill and memory updates. In particular, explain which parts of the proposed coordination are new and which adapt existing ideas to conversational feedback. A distinction based only on natural-language feedback would need care: MSCE can convert textual user feedback into a scalar update signal. This comparison should clarify the contribution rather than assume that all related methods require reference answers. A direct benchmark comparison would be informative where protocols can be aligned, but the requested conceptual distinction does not require transplanting incompatible original settings. (Lines 047–060, 089–110; MSCE, Section 3 and Appendix B.3; Mem²Evolve, Section 3.)
-
Does SkillWeave retain an advantage over a simple combination of AutoSkill and the same knowledge base? Please compare AutoSkill, AutoSkill plus knowledge retrieval, and SkillWeave on MemoryBench. For the latter two, share a frozen knowledge base constructed during training using the same permitted knowledge sources, without held-out queries or their reference answers; let the augmented AutoSkill retrieve relevant entries alongside its own skills. Use the same backbone, training logs, and held-out queries, and align retrieval access, tool interfaces, length control, and total context budget as closely as feasible. Report task scores and uncertainty, actual tokens and calls, and construction and inference costs, accounting explicitly for the shared knowledge-base cost. This would isolate the benefit of skill organization, updating, and use after controlling factual information; it would not test the knowledge-base construction method itself. (Lines 302–320; Table 2 and lines 356–374.)
-
Does grouping feedback by topic improve held-out performance beyond the effects of batching and skill capacity? Please compare topic-based grouping with random grouping while retaining the same training logs, feedback text, and knowledge base. Match group sizes, update opportunities, skill capacity, and skill-generation budget as closely as feasible, and report any remaining differences. Repeat the random grouping across seeds and report the performance difference and uncertainty. This directly tests whether semantic grouping helps combine partial feedback into reusable procedures, which the current 36-versus-8-skill comparison does not isolate. (Lines 039–045 and 232–242; Table 2 and lines 351–356.)
6 comments
Response to Reviewer VaGZ (Part 2/2: W2, Q2, Q3)
2. Controlled Comparison Against AutoSkill Sharing the Same Frozen KB (W2 & Q2)
Thank you for designing this clean protocol to isolate skill learning after controlling factual information. Following your protocol, we compared AutoSkill (AS), AutoSkill + Frozen KB (AS+KB), and SkillWeave (SW) across all three domains and their mean (MB Avg.) on DS-V4-Flash using identical training logs and held-out queries. AS+KB shares the exact frozen KB built during training (with zero access to held-out queries or references) and retrieves the top-5 entries via BM25 alongside its own skills (capped at 12,500 chars, averaging 2,298 / 1,492 / 1,701 injected tokens per query on Legal / Academic / Open-Domain), with the shared KB build cost (1.18 / 0.98 / 1.30) explicitly included in AS+KB:
| Domain | AutoSkill | AutoSkill + Frozen KB | SkillWeave | SW − (AS+KB) [95% CI] | Build Calls / Cost (AS+KB vs. SW) | Test / Total Cost (AS+KB vs. SW) |
|---|---|---|---|---|---|---|
| Legal | 41.5 | 46.1 | 50.1 | +3.9 [+1.0, +6.9] () | 2,640+KB (5.49) vs. 996 (1.60) | 0.66 / 6.15 vs. 0.25 / 1.85 |
| Academic | 67.1 | 67.6 | 68.8 | +1.2 [-0.8, +3.2] () | 3,320+KB (5.78) vs. 1,357 (1.33) | 0.29 / 6.07 vs. 0.40 / 1.73 |
| Open-Domain | 63.0 | 63.3 | 74.6 | +11.3 [+7.6, +15.1] () | 11,816+KB (6.96) vs. 1,591 (1.69) | 0.36 / 7.32 vs. 0.51 / 2.21 |
| MB Avg. | 57.2 | 59.0 | 64.5 | +5.5 |
Augmenting AutoSkill with the exact same curated KB improves Legal (though still trailing KB-free SkillWeave w/o KB at 47.1), yet SkillWeave retains a substantial lead across domains (+5.5 average) at under one-third of the total cost. This isolates the value of inducing procedures from full query, response, and feedback triples (whereas AutoSkill extracts from user queries alone) and invoking KB search on demand via skill directives rather than unconditionally prepending retrieved passages onto non-factual writing tasks.
3. Topic Grouping vs. Capacity- and Budget-Matched Random Grouping (W2 & Q3)
Thank you for noting that comparing 36 versus 8 skills conflates topic grouping with batching and skill capacity. On DS-V4-Flash Legal, holding training logs, feedback text, and the frozen KB identical, we ran 3-seed random-grouping controls: within each batch (), clustering and topic-extraction calls run as normal to match call budgets, after which newly assigned cases have their group labels randomly permuted across groups and receive neutral identifiers (Group 1, Group 2, ...) so mismatched titles do not mislead generation. This matches per-batch group-size distributions, update opportunities, and skill capacity. We also ran a 3-seed control keeping semantic grouping with neutral identifiers:
| Setting (DS-V4-Flash Legal) | Score | #Skills | [95% CI] |
|---|---|---|---|
SkillWeave (semantic grouping + explicit topic names) | 50.1 | 8 | ref. |
| Random grouping + neutral IDs (3 seeds) | 48.1 ± 2.7 | 7 to 9 | −2.0 [-5.8, +1.9] |
w/o clustering (log-by-log updates) | 47.8 | 36 | −2.2 |
| Semantic grouping + neutral IDs (3 seeds) | 45.8 ± 0.5 | 3 to 8 | −4.3 [-7.3, -1.3] () |
Single topic (one coarse topic per cluster) | 45.0 | 8 | −5.1 |
(Per-batch permutation matches within-batch group sizes, while across 20 batches cross-batch topic merging yields a minor natural difference of 7 to 9 vs. 8 final skills.)
Together with Table 2, these controls clarify how grouping works. While batching into compact libraries avoids the fragmentation of 36 single-log skills, random grouping lowers mean score and introduces severe seed-to-seed instability (±2.7, far exceeding SkillWeave's retrain spread in Appendix F), showing that aggregating semantically related remarks is essential for stable induction. Moreover, the significant drop under Semantic grouping + neutral IDs demonstrates that extracting explicit task-type names and descriptions provides indispensable semantic anchors for defining genre boundaries and BM25 catalog retrieval, consistent with the sharp drop of Single topic when distinct sub-genres are merged under one broad label. We have added both controlled studies to Appendix L.3 (Table 17) and Appendix L.4 (Table 18).
Thank you again for these constructive experimental designs, which substantially sharpened both our positioning and our empirical attribution.
Thank you for the additional experiments and the work you have put into addressing my comments. The shared-KB comparison and the clearer positioning have addressed much of my concern. I would like to see one further control: random grouping with explicit topic names generated from the randomized groups, using the same naming and retrieval procedure as full SkillWeave, with comparable skill capacity and update budgets. Please report the score difference and its uncertainty over the same three seeds. This would help distinguish the effect of semantic grouping from that of naming. Once this comparison is included, I would be willing to raise my rating to 3.
Response to Reviewer VaGZ (Part 1/2: W1, Q1)
We sincerely thank you for recognizing the practical value of separating components, and for proposing clear positioning dimensions and rigorous controlled experiments. We address W1/Q1 here and W2/Q2 and Q3 in Part 2/2.
1. Positioning Within Skill and Memory Co-Evolution: MSCE and Mem²Evolve (W1 & Q1)
Thank you for highlighting these concurrent preprints. We have refined Section 1, Section 2.2, and Appendix I (Table 9) to position SkillWeave explicitly alongside MSCE and Mem²Evolve across the five dimensions you raised:
Avoid when) + factual KBMaintaining complementary stores, aggregating across multiple episodes before inducing procedures (such as MSCE's nmin episodes or SkillOpt's minibatches), and tracking applicability boundaries are shared principles across recent self-evolving agent work, which we explicitly credit in Section 2.2 and Appendix I. Notice also the difference in feedback protocol and setting: MSCE's main experiments on multi-step tool environments (
EvoAgentBench) rely on environment verifier rewards, and its simulated textual feedback study (their Appendix 10.2) invokes a coach model on verifier-flagged failed trajectories with access to full internal traces, verifier results, and stored reflections before quantifying the diagnosis into a scalar return Ri (because both MSCE and Mem²Evolve operate over online multi-step action-observation trajectories rather than static dialogue logs, and Mem²Evolve has no public code, their native protocols do not directly transplant to offline conversational logs). More fundamentally, converting textual feedback into a scalar Ri effectively governs whether to promote or retire a policy, but scalar compression discards which specific domain fact was missing and how a procedure should be rewritten. For offline user logs with no verifier or re-execution rollouts, SkillWeave introduces (1) evidence-grounded gap distillation against an external reference corpus to prevent unverified claims or hallucinations from entering long-term memory, and (2) softAvoid whenself-judgment in a progressive-disclosure catalog excluded from BM25 scoring, preventing the severe false drops caused by lexical hard filtering.