- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies skill self-evolution for LLM customer-support agents, arguing that single-turn QA feedback stops exposing new defects after the first revision. The authors propose SkillMuse, which reconstructs a constrained simulated user from each human-handled ticket (intent agenda, behavior facts, emotion trajectory, gated by an intent state machine), verifies multi-turn trajectories against the human solution, forwards only failures attributed to "Knowledge Gap" to a bounded skill editor, and adds a governance layer that checks fact consistency against the initial and previous skill and flags structural bloat or broken references. On nine production skills from one cloud provider, SkillMuse reaches 81.8% TSR after four rounds versus 66.4% for SkillForge-style single-turn evolution and 58.8% for self-reflectiom.
Critical strengths
-
The setting with held-out and chronologically split evaluation is realistic and reasonable. The paper evolves nine production skills, including 98 reference files and 2,000 escalated tickets, splits tickets chronologically into development and evaluation sets, and selects versions on the development set only. This is reasonable because skill-evolution work is often evaluated on the tickets it was evolved on; a temporal split is the right protocol for a deployment claim.
-
The user simulator is built from real tickets and checked by humans. Each simulated user is rebuilt from a real human-handled ticket, including the user's requests, known facts, and emotion changes, and an intent state machine ends the dialogue only after all requests have been raised and answered. Two domain experts blindly compared 200 simulated dialogues with the real tickets and reported 95.3% agreement. This is useful because the whole method depends on whether the simulated user behaves like a real one, and this check gives direct human evidence for it instead of relying only on the LLM judge.
Critical weaknesses
-
The major concern is that the evaluation protocol may favour SkillMuse by construction, and the comparison appears to be a single run. TSR on the evaluation set is computed by running the same multi-turn User Agent and the same Verifier that SkillMuse uses to generate its feedback on the development set. SkillMuse is therefore optimized against the evaluator used at test time, whereas the SkillForge baseline is optimized with single-turn QA and then judged in the multi-turn protocol. Part of the 15.4-point gap may thus reflect a match between training feedback and test protocol rather than better skills. In addition, it is unclear whether the experiments are run with different seeds and whether significance tests are conducted.
-
Another concern is that the key claim of the paper, i.e., multi-turn dialogue can expose defects that single-turn QA cannot, is not directly verified. The paper does not show how many of the failure signals in later rounds actually come from the second or later turns of a dialogue. In fact, in the paper's own case study, the error is already visible in the first exchange. Since the main difference from SkillForge is the feedback source, and using a user simulator to train dialogue systems is not new, this claim is what the novelty mainly relies on. Without such an analysis, SkillMuse looks more like SkillForge with a multi-turn simulator plugged in. Besides, the ablation study only has two variants, and other components, e.g., attribution filtering, the intent state machine, and the dual-anchor check, are not tested separately. The regression rate is only reported with governance, so it is hard to tell how much governance actually helps. The paper also claims that the evolution is faster and cheaper than manual curation, but no token or time cost is reported. The authors are encouraged to address these issues.
Questions for authors
- Have the authors evaluated all methods with a different simulator or judge, or with human evaluation, instead of the same simulator and Verifier used to generate SkillMuse's feedback?
- Are the results in Table 2 obtained from a single run? Have the authors run the experiments with different seeds?