- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SkillCertainty, a framework that combines trajectory-level routing with step-level conformal diagnostics to guide revision of an LLM agent's skill document. In ATUQ, ten executions with the existing skill and ten without are collected per task. Successful with-skill executions are used to build a position-stratified StepLibrary of per-bucket conformal thresholds and a Skill Reference Vector, and failed without-skill executions serve as contrastive negatives for learning action-importance weights. In CSASO, each new trajectory receives a composite score of task outcome and adherence to the Skill Reference Vector, and a fixed threshold on this score routes it. Conforming trajectories are distilled into a skill that is validated in one fresh run. Nonconforming trajectories go through Step-CP diagnostics, which flag steps that exceed their bucket threshold and insert them into the skill-generation prompt. On SkillsBench, across six LLM backends and four skill-evolution baselines, the method reports competitive final scores, the lowest adherence nonconformity, and fewer evolution iterations than EvoSkill.
Critical strengths
-
Step-level diagnostics give the skill generator targeted feedback. Step-CP diagnostics highlight specific failed or unusual steps and their outputs alongside the task outcome and raw trajectory. Table 3 shows that adding these diagnostics raises mean ΔGS from +0.206 to +0.384 on three tasks compared with using raw trajectories alone. The per-task examples in the appendix illustrate how flagged steps inform specific skill revisions.
-
Fresh validation of distilled skills. Table 4 shows that the score of the trajectory triggering distillation can overstate the score obtained when the resulting skill is executed: mean ΔGS falls from 0.293 at detection to 0.094 on a fresh run, and Energy Market Pricing falls from 1.000 to 0.000. This is a useful finding for skill-evolution methods that judge a skill by the trajectory it was distilled from.
Critical weaknesses
-
The benefit of conformal calibration is unclear. Table 3 shows that adding Step-CP diagnostics improves results compared with raw trajectories alone. However, it does not compare them with simpler feedback, such as listing failed steps and their outputs or highlighting rare actions without calibrated thresholds. Algorithm 1 already prioritizes failed-status steps, and Appendix B acknowledges that the calibration does not establish predictive coverage for arbitrary future steps. The experiments therefore show a benefit from the diagnostic feedback, but do not establish whether conformal calibration is needed for that benefit.
-
The adherence and efficiency comparisons have limitations. SkillCertainty uses s_adh both to evaluate adherence and to decide when to stop revising the skill. The baselines do not explicitly target this score, so lower s_adh alone does not demonstrate better task performance or transfer. The efficiency comparison reports evolution iterations but excludes the 20 ATUQ runs per task and the fresh validation run. The main-comparison update budgets are unspecified, and it is unclear whether the baselines receive the same ATUQ runs. The method uses fewer updates than EvoSkill, but the paper does not show whether it uses fewer task executions or tokens overall.
-
Skills are not tested on new tasks. The paper motivates skills as reusable, but evaluates them on the same tasks used during skill evolution. No new tasks or task variants are tested, so the results do not show whether the evolved skills can be reused beyond those tasks.
Questions for authors
- How does Step-CP compare with simply highlighting failed steps and their outputs, using the same skill generator and budget?
- Why do Tables 6 (δ = 0.25) and 7 (α = 0.10) imply different starting scores for GPT-5.1? Why do four EvoSkill composite scores in Table 1 differ from Eq. 11 applied to the reported GSf and s_adh?