- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates whether providing the same procedural knowledge to an LLM Agent as a Skill or as ordinary context affects its execution behavior. The authors propose PAT-PV to examine the effects of Skills on execution errors, information verification, and task success rates by controlling factors such as content, source labels, and formatting. They also introduce a verification mechanism based on assertion–action dependencies. Experimental results show that loading incorrect Skills increases critical execution errors, while PAT-PV can reduce these errors to some extent.
Critical strengths
-
The research question is novel. While existing studies mainly focus on whether Skills improve Agent performance, this paper investigates whether the way Skills are loaded affects Agent behavior.
-
The experiments are relatively comprehensive, using multiple evaluation metrics and different types of experiments to analyze the results.
Critical weaknesses
-
In Appendix B.2 (page 16), the random seeds are {11, 23, 37, 53, 71}, but the values reported in Table 13 of the appendix are completely different.
-
The reported pass@k results contain clear errors. Appendix B.2 defines pass@k as the proportion of tasks that succeed at least once across k independent runs. By this definition, pass@5 should not be lower than pass@1, yet the paper consistently reports lower pass@5 values.
-
Skill and plain context differ not only in their registration status but also in their status descriptions (loaded/provided). Therefore, the influence of explicit status cues cannot be ruled out, and the observed differences cannot be attributed solely to Skill registration.
Questions for authors
The authors are requested to address the weaknesses raised above and make the necessary revisions or conduct additional experiments.