- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to extract useful skills from the execution records of intelligent agents. The author compared using only the success records, using both success and failure records with result labels, and the same batch of records without labels, as well as comparing step-by-step workflows and declarative operational ontologies. Experiments checked the effectiveness of these configurations on tasks not involved in skill construction and analyzed whether the skills were read and why they still failed after reading.
Critical strengths
S1. The label comparison is persuasive. Fixing the same batch of execution records and changing the result labels and accompanying instructions makes it easier to see the effect of this configuration and avoid confusing it with a different batch of records.
S2. The evaluation separates skill construction from testing tasks, and also uses repeated runs and paired comparisons for the same task. This is more capable of demonstrating whether the skills are useful than only showing a few successful cases.
S3. The author checked whether the skills were read and whether they still failed after reading. These analyses can help distinguish problems with the skill content and actual usage issues, and also make negative results more interpretable.
Critical weaknesses
W1. I think this paper is more about comparing various approaches to generating skills. The experiment has some value, but there is not much novelty in the method. I still don't quite understand how these findings can help us improve the skills.
W2. Workflow is a global skill, but the ontology is generated separately for each task group, and the organizational methods on both sides are different. Although the paper mentioned this difference, now seeing the different effects, I'm not sure whether it's the representation form that matters, or the organizational method.
W3. Regarding the differences from existing work, I think it's necessary to explain clearly. "AWM" has already summarized reusable processes from the execution records, and "From Raw Experience to Skill Consumption" and "Demystifying Agent Skills" have also studied how to use success and failure records. This paper adds the comparison of workflow and ontology on enterprise tasks, but it's not clear yet what new key discoveries these experiments have brought.
W4. I looked at the prompt in the appendix and found that these prompts for generating skills actually provide a very specific operational framework, and it leans more towards business process tasks. Although the author also tested other types of tasks, I will still be concerned about how much the results depend on how well this framework matches the tasks. However, I haven't seen a control where skills are generated based solely on domain and tool information without considering the execution records. Therefore, it's still difficult to determine exactly how much additional improvement has been achieved from the experiences within the trajectory.
Questions for authors
Q1. Can you please specifically explain what the most crucial new discovery of this paper is compared to the works mentioned above, and how it can help us improve the skills?
Q2. Can you use the same set of records to generate skills in both forms, using the same task grouping, and then compare the effects using the same loading method?
Q3. Can you add a control group that generates skills without looking at the execution records, only based on domain descriptions and tool information, and test it on both business process tasks and analytical tasks? This way, I can see how much additional improvement the experience in the trajectory brings.
2 comments
Thank you for the response. The paper still lacks comparisons using the same task grouping and generating skills without execution records. These are important for understanding where the gains come from, so I will keep my score.
Response to 3y8p: Practical Takeaways, Relation to Prior Work, and Prompt Framework
Thank you for the review. We are glad you found the label comparison persuasive, the separation of skill construction from testing with repeated paired runs more informative than showing a few successful cases, and the read-versus-failure analyses useful for interpreting negative results.
[W1, Q1] "How do the findings help us improve skills?" The findings translate into concrete choices:
[W3, Q1] Relation to AWM, Raw Experience to Skill Consumption, and Demystifying Agent Skills. Those works cover mining routines and some evidence and label effects, mostly with the evidence pool varying across conditions. Here the trajectories are held fixed while label visibility changes, and the setting is exact-state enterprise tasks. We also report routing-versus-execution failures for each condition. The paper adds these measurements to the existing picture and does not claim a new mining method.
[W2, Q2] Form vs. organization. Both forms are mined from the same fold's training pool and loaded as native skills with the same routing. Per skill, the miner sees at most ten tasks with five trajectories each. An ontology is organized by task group by nature, so having several skills is part of the form. We thank you for suggesting a scoped workflow on identical task groups, which would separate form from organization; we regard it as a valuable next step.
[W4, Q3] Dependence on the prompt framework; a no-trajectory control. The prompts were authored against ThinkingBox-Bench and reused unchanged for APEX-Agents, which has analytical deliverables, and gains there are smaller. So we agree that framework fit matters, and it is one reason the paper recommends tailoring meta-skills. The framework is fixed across arms, yet changing only the trajectories produces large differences, for example Success vs. Goldilocks in retail and hotel (4.5 and 4.9 pp) and labeled vs. blind in ThinkingBox-Bench (3.1 pp). The framework alone cannot explain these. A control that generates skills from domain and tool descriptions without trajectories would need a different meta-skill. We thank you for suggesting it; it is an interesting direction, but we expect it to add little production insight, since a deployment would compare the mined configurations that are actually available to it. The no-skill baseline gives the total lift, but we have not run the no-trajectory control, so we cannot yet quantify the added value of trajectory experience.
Summary: We clarified the practical implications, how the work relates to prior studies, and what the prompt-framework concern does and does not affect. We hope this addresses your concerns, and we would be glad to answer follow-up questions.